{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Otto商品分类——线性SVM\n",
    "\n",
    "我们以Kaggle 2015年举办的Otto Group Product Classification Challenge竞赛数据为例，分别调用\n",
    "缺省参数LinearSVC、\n",
    "LinearSVC + CV进行参数调优（手动实现循环）。\n",
    "\n",
    "Otto数据集是著名电商Otto提供的一个多类商品分类问题，类别数=9. 每个样本有93维数值型特征（整数，表示某种事件发生的次数，已经进行过脱敏处理）。 竞赛官网：https://www.kaggle.com/c/otto-group-product-classification-challenge/data\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 首先 import 必要的模块\n",
    "import pandas as pd \n",
    "import numpy as np\n",
    "\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "from matplotlib import pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 准备数据 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# 读取数据，只用tf_idf特征\n",
    "train = pd.read_csv(\"./Otto_FE_train_tfidf.csv\")\n",
    "\n",
    "# drop ids and get labels\n",
    "y_train = train['target']   #形式为Class_x\n",
    "X_train = train.drop([\"id\", \"target\"], axis=1)\n",
    "\n",
    "\n",
    "#保存特征名字以备后用（可视化）\n",
    "feat_names = X_train.columns \n",
    "\n",
    "#sklearn的学习器大多支持稀疏数据输入，模型训练会快很多\n",
    "from scipy.sparse import csr_matrix\n",
    "X_train = csr_matrix(X_train)\n",
    "\n",
    "# 用train_test_split，从将数据集中随机抽取10000条记录来训练模型\n",
    "from sklearn.model_selection import train_test_split\n",
    "X_train_selected, X_abandon, y_train_selected, y_abandon = train_test_split(X_train, y_train, train_size = 10000,random_state = 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(10000, 93)\n"
     ]
    }
   ],
   "source": [
    "print (X_train_selected.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 模型训练"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 线性SVM正则参数调优"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "线性SVM 采用GridSearchCV进行交叉验证调优，需要调整正则超参数包括C（正则系数，一般在log域（取log后的值）均匀设置候选参数）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=None, error_score=nan,\n",
       "             estimator=SVC(C=1.0, break_ties=False, cache_size=200,\n",
       "                           class_weight=None, coef0=0.0,\n",
       "                           decision_function_shape='ovr', degree=3,\n",
       "                           gamma='scale', kernel='linear', max_iter=-1,\n",
       "                           probability=True, random_state=None, shrinking=True,\n",
       "                           tol=0.001, verbose=False),\n",
       "             iid='deprecated', n_jobs=4,\n",
       "             param_grid={'C': array([1.e-01, 1.e+00, 1.e+01, 1.e+02, 1.e+03])},\n",
       "             pre_dispatch='2*n_jobs', refit=True, return_train_score='warn',\n",
       "             scoring='neg_log_loss', verbose=0)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#线性SVM\n",
    "\n",
    "from sklearn.svm import SVC\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "\n",
    "C_s = np.logspace(-1, 3, 5)# logspace(a,b,N)把10的a次方到10的b次方区间分成N份 \n",
    "param_grid={'C':C_s}\n",
    "\n",
    "#这里用neg_log_loss来评估模型性能，熵是由概率计算得到的，所以probability=True\n",
    "grid = GridSearchCV(SVC(kernel='linear',probability=True), param_grid, n_jobs = 4,return_train_score='warn',scoring='neg_log_loss')\n",
    "grid.fit(X_train_selected,y_train_selected)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best score: 0.634582 using {'C': 1.0}\n",
      "[0.66625262 0.63458173 0.63986723 0.66665728 0.69228174]\n",
      "[0.63899622 0.57591121 0.54425827 0.55823614 0.58478772]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEGCAYAAAB/+QKOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3dd3xUddb48c9JBxJqQg2QgIAgIEqkN11hAQtWVMS6guXns+quPo+u7rq667Pq7rOrrm0BsSsqdkFFRQQpQlCULjUk1BBqQgIp5/fHvYExDCGB3Nxk5rxfr3ll5pbJmYHMmfs93yKqijHGGFNWhN8BGGOMqZksQRhjjAnKEoQxxpigLEEYY4wJyhKEMcaYoKL8DqCqJCYmakpKit9hGGNMrbJ48eKdqpoUbF/IJIiUlBTS09P9DsMYY2oVEck41j5rYjLGGBOUJQhjjDFBWYIwxhgTVMjUIIIpLCwkKyuLgoICv0PxXFxcHMnJyURHR/sdijEmRIR0gsjKyiIhIYGUlBRExO9wPKOq5OTkkJWVRWpqqt/hGGNCREg3MRUUFNCkSZOQTg4AIkKTJk3C4krJGFN9QjpBACGfHEqFy+s0xlSfkE8QxhgTqoqKS/j4xy28uXCTJ89vCcJje/bs4dlnnz2hc5944gkOHDhQxREZY2q7/QWFTJqznsF/n8V/vfkDb6dn4sXaPpYgPGYJwhhTVbbsyed/p6+k399m8tdpK2nVqA4Tr03j3Vv6edLMHNK9mGqCe++9l3Xr1tGjRw+GDh1K06ZNefvttzl48CAXX3wxDz30EHl5eYwePZqsrCyKi4v54x//yPbt29myZQtnn302iYmJfP31136/FGOMT5Zt3svEOeuZ9tNWFBjRtTnjBrbj9NYNPf29YZMgHvp4OSu27KvS5+zSsj4PXnBaucc8+uijLFu2jCVLljBjxgymTp3KwoULUVUuvPBCZs+eTXZ2Ni1btmTatGkA7N27lwYNGvDPf/6Tr7/+msTExCqN2xhT85WUKLN+3sHE2RuYvz6H+NgoruuXwg39U0huVLdaYgibBFETzJgxgxkzZnDGGWcAkJuby5o1axg4cCB33303//M//8P555/PwIEDfY7UGOOXgsJi3v9hM5PmrGdddh4tGsTxh5GncmWvNtSPq96BsGGTII73Tb86qCr33XcfN99881H7Fi9ezPTp07nvvvsYNmwYf/rTn3yI0Bjjl5zcg7y6IINX52eQk3eIrq3q8+SVPRjZrQXRkf6Ui8MmQfglISGB/fv3A/DrX/+aP/7xj1x99dXEx8ezefNmoqOjKSoqonHjxowdO5b4+HheeumlX5xrTUzGhK512blMmrOB977P4mBRCb86tSk3DWxHn3aNfR/fZAnCY02aNKF///507dqVESNGMGbMGPr27QtAfHw8r732GmvXruWee+4hIiKC6OhonnvuOQDGjx/PiBEjaNGihRWpjQkhqsp3G3Yxac56vly5g5ioCC49sxW/GZDKKU0T/A7vMPGi76wf0tLStOyCQStXrqRz584+RVT9wu31GlPbFBaXMH3pVibN2cDSzXtpXC+Ga/q05Zq+bUmMj/UlJhFZrKppwfbZFYQxxnhsf0EhUxZm8uLcDWzZW0C7pHr878XduOTMVsRFR/od3jFZgjDGGI9s3pPPS3M38ObCTHIPFtE7tTF/uagrZ3dqSkREzZ8/zRKEMcZUsaVZ7sC2pVsBOK9bC8YNbEe35AY+R1Y5liCMMaYKlJQoM1ftYOKc9Xy3YRfxsVHc2D+F6/un0qphHb/DOyGeJggRGQ48CUQCk1T10SDHjAb+DCjwo6qOcbc/DpyHM1/UF8AdGioVdWNMyCgoLObd77N44dsNrM/Oo2WDOO4f2ZkrerWu9oFtVc2zBCEikcAzwFAgC1gkIh+p6oqAYzoA9wH9VXW3iDR1t/cD+gPd3UO/BQYDs7yK1xhjKmNn7kFemZ/Bawsy2JV3iG6tGvg+sK2qeXkF0QtYq6rrAURkCjAKWBFwzDjgGVXdDaCqO9ztCsQBMYAA0cB2D2P1zJ49e3jjjTe47bbbKnXeyJEjeeONN2jY0NvJuIwxlbN2Ry4vfLued7/fzKGiEs7t7Axs653q/8C2quZlgmgFZAY8zgJ6lzmmI4CIzMVphvqzqn6mqvNF5GtgK06CeFpVV3oYq2dKp/sumyCKi4uJjDx297bp06d7HZoxpoJUlfnrc5g0ZwMzV+0gNiqCy3om85sBqbRPivc7PM94mSCCpdKyNYQooAMwBEgG5ohIVyAR6OxuA/hCRAap6uxf/AKR8cB4gDZt2lRd5FUocLrv6Oho4uPjadGiBUuWLGHFihVcdNFFZGZmUlBQwB133MH48eMBSElJIT09ndzcXEaMGMGAAQOYN28erVq14sMPP6ROndpZ9DKmNikd2DZxznqWbd5Hk3ox3HluB67p05YmPg1sq05eJogsoHXA42RgS5BjFqhqIbBBRFZzJGEsUNVcABH5FOgD/CJBqOoEYAI4I6nLjebTe2Hb0hN9LcE17wYjjqq7/0LgdN+zZs3ivPPOY9myZaSmpgIwefJkGjduTH5+PmeddRaXXnopTZo0+cVzrFmzhjfffJOJEycyevRo3n33XcaOHVu1r8UYc9i+gkKmLNzEi3M3snVvAe2T6vG3S7px8Rk1e2BbVfMyQSwCOohIKrAZuBIYU+aYD4CrgJdEJBGnyWk90A4YJyJ/w7kSGQw84WGs1aZXr16HkwPAU089xfvvvw9AZmYma9asOSpBpKam0qNHDwB69uzJxo0bqy1eY8JJ1u4DvDh3I28tcga29W3XhEcu7sqQjrVjYFtV8yxBqGqRiNwOfI5TX5isqstF5GEgXVU/cvcNE5EVQDFwj6rmiMhU4BxgKU6z1Geq+vFJBXScb/rVpV69eofvz5o1iy+//JL58+dTt25dhgwZQkFBwVHnxMYeuZSNjIwkPz+/WmI1Jlz8mLmHiXPW8+mybQhwfvcW3DSwHV1b1a6BbVXN03EQqjodmF5m258C7ivwO/cWeEwxcPSiCbVQ4HTfZe3du5dGjRpRt25dVq1axYIFC6o5OmPCV0mJ8uXK7Uyas4GFG3eREBvFTQNSua5fCi1r6cC2qmYjqT0WON13nTp1aNas2eF9w4cP5/nnn6d79+506tSJPn36+BipMeEh/9CRgW0bdubRqmEdHjivM1ec1ZqEWj6wrarZdN8hJNxerzGVkb3/IK/O38irCzLYfaCQ05MbcNPAdozo2pyoEBnYdiJsum9jTNhas30/k+Zs4P0lmyksLuHczs24aUAqvUJwYFtVswRhjAk5qsq8dTlMnLOeWauziY2K4HJ3YFu7EB7YVtVCPkGoalh8SwiVpkJjTkZhcQmf/LSFibM3sGLrPhLjY/jd0I6M7dOWxvVi/A6v1gnpBBEXF0dOTg5NmjQJ6SShquTk5BAXF+d3KMb4Ym9+IW8u3MRLczeybV8BpzSN57FLuzGqR3gNbKtqIZ0gkpOTycrKIjs72+9QPBcXF0dycvLxDzQmhGTuOsDkuRt4e1EmeYeK6de+CX+7pBuDOyaF5cC2qhbSCSI6OvoXo5aNMaHhh027mTRnA58u20qECBec3pKbBqZyWsvwHthW1UI6QRhjQkfx4YFt61m0cTcJcVGMG9SO6/ul0KKBDWzzgiUIY0yNln+omKmLM3nh2w1szDlAq4Z1+NP5XRh9VmviY+0jzEv27hpjaqQd+wt4ZV4Gr32XwZ4DhZzeuiFP/7oTw08L74Ft1ckSBOHTFdaY2mDN9v1MnLOeD37YQmFJCUM7N2PcoHaktW1kf6fVLOwTREFhMVdMWMCVZ7VmdFprIq3ngzG+WJyxm+dmrePLlduJi47girNac+OAVFIT6x3/ZOOJsE8QOXmHiI4Q7ntvKS/N3ch9I09lcMck+6ZiTDVQVb75OZvnZq3juw27aFg3mjvP7cC1fVNsYFsNENKT9VWUqvL58m08+ukqNuYcYGCHRO4b0ZkuLetXcZTGGHB6JE1fupXnZq1jxdZ9tGgQx00D23HlWa2pZ4XnalXeZH2WIAIcKirhtQUZPDVzDXvzC7m8ZzK/H9aJZvVthLIxVaGg0Jlqe8Ls9WTkHKBdUj1uGdyei3q0IibKCs9+sARRSXsPFPL012t4eV4GkRHCuEHtuHlQO/tmY8wJ2l9QyOvfbeKFbzeQvf8gpyc34NYhpzCsSzMb8ewzSxAnaFPOAR7/fBWf/LSVpIRYfj+0I5dbIduYCtuZe5AX527glfkZ7C8oYsApidw2pD1924f2/Gi1iW8JQkSGA0/irEk9SVWPWhhaREYDf8ZZe/pHVR3jbm8DTAJau/tGqurGY/0uLxJEqe837eaRaStZnLGbTs0SuG/kqQzp1NST32VMKMjcdYCJc9bz1qJMDhWXMKJrc24Z3J7uyQ39Ds2U4UuCEJFI4GdgKJAFLAKuUtUVAcd0AN4GzlHV3SLSVFV3uPtmAY+o6hciEg+UqOqBY/0+LxMEOIXsz5Zt49HPVpHhFrL/MLIznVtYIduYUqu27eP5Wev4+KetRAhcckYy4we3o72twVBj+bWiXC9graqud4OYAowCVgQcMw54RlV3AwQkhy5AlKp+4W7P9TDOChERRnRrwa86N+PVBRk89dUaRj41xwrZxgDpG3fx3Kx1fLVqB3VjIrmhXwo3DWxH8wb2d1GbeZkgWgGZAY+zgN5ljukIICJzcZqh/qyqn7nb94jIe0Aq8CVwr6oWB54sIuOB8QBt2rTx4jUcJSYqgt8MSOWyM5P598w1vDx/Ix//uJXxg9ox3grZJoyoKrNWO2MYFm7cRaO60fxuaEeu7duWhnVtDEMo8PLTLFgFqmx7VhTQARgCJANzRKSru30gcAawCXgLuB544RdPpjoBmABOE1PVhX58DepG88D5Xbi2bwqPfb6KJ79awxsLN1kh24S8ouISprljGFZt20/LBnE8eEEXrjirNXVj7AtSKPHyXzMLp8BcKhnYEuSYBapaCGwQkdU4CSML+CGgeeoDoA9lEkRN0KZJXZ4ZcyY39t/N/05fyb3vLeXFuRv5w3mdGdwxye/wjKkyBYXFTF3sjGHYtOsApzSN5x+Xn86oHi2JtsnzQpKXCWIR0EFEUoHNwJXAmDLHfABcBbwkIok4TUvrgT1AIxFJUtVs4BzAuwp0FejZthFTb+nLp8ucEdnXTV7IwA6J3H9eZ05tboVsU3vtKyjktQUZTP52IztzD9KjdUMeOK8z53a2MQyhzrMEoapFInI78DlOfWGyqi4XkYeBdFX9yN03TERWAMXAPaqaAyAidwNfidNZejEw0atYq4qIMLJbC37VuSmvLdjkFLKfnMPlPVvzu2EdrZBtapUd+wt4ce5GXpufwf6DRQzskMhtQ86gT7vGNoYhTNhAOQ/tOXCIp2eu5eX5G4mKiLBCtqkVNuUcYMKcdbydnkVhcQkju7Xg1sHt6drKlvMMRTaS2mcZOXk8/tlqpi3dStOEWH4/rCOX9bRCtqlZVm7dx/PfrOPjH7cQFRHBpT1bMX5Qe5tuO8RZgqghFmfs5pFpK/h+0x5ObZ7AfSOtkG38t2jjLp79ei1fr86mXkwkV/dpy28GpFqTaJiwBFGDqCrTl27jsc9WsWnXAQZ1TOIPI0+1QrapVqrKzFU7eG7WOtIzdtO4Xgw39Evh2r4pNKgb7Xd4phpZgqiBDhYV8+r8DP49cy37Cwq5vGdrfj+sI03tW5vxUFFxCZ/85IxhWL19P60a1mH8oHaMTmtNnZhIv8MzPrAEUYPtOXCIf89cyyvznUL2zYOdQrYNODJVqaCwmHfSM/nP7PVk7c6nQ9N4bh3SngtOtzEM4c4SRC1ghWzjhb35pWMYNpCTd4gz2jTktiGn8KtTm9oYBgNYgqhVFmfs4q/TVvKDW8j+w8jODLJCtqmkHfsKeGHuBl5fsIncg0UM7pjErUPa0zvVxjCYX7IEUcuUFrIf/WwlmbvyGdQxiftHdqZT8wS/QzM1XEZOHv+ZvZ6pi7MoKh3DMKQ9p7W0MQwmOEsQtVRpIfupr9aQe7CI0Wmt+d1QK2Sboy3fspfnv1nPtJ+cMQyXpSUzfmA7UmwMgzkOSxC1XGAhOzryyIhsK2SHN1Vl4YZdPDtrHd/8nE18bBRX92nDb/qn2pcIU2GWIELExp15PP75KqYv3UbThFjuHtaJS3smWyE7zJSUKF+t2sFzs9by/aY9NKkXw40DUhnbpy0N6tgYBlM5liBCTNlC9v3ndWZgBytkh7rC4hI+/nELz3+zjp+359KqYR1uHuyMYYiLtjEM5sRYgghBqsq0pVt57LNVZO7KZ3DHJP5gheyQlH+omLfTM5kwez2b9+TTqVkCtw5pz3ndW9gYBnPSLEGEsINFxbwyL4N/z3QK2Vec1Zq7zrVCdijYe6CQV+Zv5MV5G9mVd4iebRtx25D2nN3JxjCYqmMJIgzsznMK2a8ucArZNw9qz7hBqVbIroW27yvghW838PqCDPIOFXN2pyRuHXIKvVIb+x2aCUGWIMLIxp15PPbZKj5dto1m9WP5/bBOXHqmFbJrgw0785gwex3vLt5MUUkJ53dvyS2D29OlpU3kaLxjCSIMpW90CtlLMq2QXdMt27yX575Zx6dLtxIVGcHotGTGD2xPmyZ1/Q7NhAFLEGFKVfnkJ6eQnbXbCtk1iaqyYP0unp21ljlrdpIQG8XYvm25oX8KTROsfmSqj28JQkSGA0/irEk9SVUfDXLMaODPgAI/quqYgH31gZXA+6p6e3m/yxLEsQUtZA/taB9EPigpUb5YuZ3nZq1jSeYeEuOPjGGoH2djGEz18yVBiEgk8DMwFMgCFgFXqeqKgGM6AG8D56jqbhFpqqo7AvY/CSQBuyxBnLzdeYd4auYaXp2fQUxUBLcMbs9NA62QXR0Ki0v4cIkzhmHtjlxaN67D+EHtubxnso1hML4qL0F4+cnQC1irquvdIKYAo4AVAceMA55R1d0AZZJDT6AZ8BkQNHhTOY3qxfDgBadxbd8UHv9sFf/84mde/y7DCtlV6FBRCTtzD5K937ntzD3Ilj35TF2cxZa9BZzaPIEnr+zBed1aEGVjGEwN52WCaAVkBjzOAnqXOaYjgIjMxWmG+rOqfiYiEcD/AdcAvzrWLxCR8cB4gDZt2pxYlKrww2vQ5UKIC48ZL1MT6/Hc2J4scgvZ/z31J16cu5H7R3ZmQIdEv8OrcYpLlJy80g/8Q4c//EsTQPb+g2S7P/fmFwZ9jl4pjXnk4m4M6ZRk022bWsPLBBHsr6Bse1YU0AEYAiQDc0SkKzAWmK6qmeX9ManqBGACOE1MJxTlzjXwyZ2w9G24eipExZ7Q09RGZ6U05oPb+h0uZI994TuGdHIK2R2bhXYhW1XZc6CQ7NyD7Az4gM8uc39n7kF25R2iJMj/rnoxkSQlxJKUEEuHpvH0a9+EpHjncaL7MykhlibxMcRGWTOSqX28TBBZQOuAx8nAliDHLFDVQmCDiKzGSRh9gYEichsQD8SISK6q3lvlUSZ1hFHPwvvj4b3xcNlkiAifP2YR4YLTWzK0SzNemb+Rf89cy/AnZnPFWW24a2iHWlXIVlVyDxaV+ZZfcORx7pEP/Z25ByksPvpTPyYqgqT4WBITYkluVJcz2jRyPujjYw5/4CfGO7d6sVa7MaHNyyJ1FE6R+lfAZpwi9RhVXR5wzHCcwvV1IpII/AD0UNWcgGOuB9I8L1LP+zfMeAB6jYcRj0OYNgPszjvEk1+t4bUFRwrZ4wa283VB+4LC4qM+4I9q5nG3FRSWHHV+ZITQpN6RD/jSBJAU8C2/9Bt//bgoawIyYcWXIrWqFonI7cDnOPWFyaq6XEQeBtJV9SN33zARWQEUA/cEJodq1e+/YP82mP80xDeDQXf7EobfGtWL4c8XnsZ1/VJ47NMjhey7h3XikiosZBcWl5Bz+Jt9ATv3HzqqmWene3//waKgz9G4XszhD/mebeoeSQCBTTzxsTSqG2NzFxlzAmygXKCSEnj/ZqceceHTcOY1VRNcLbZwwy4embaCH7P20rlF/XIL2SUlyq4Dh477LT97/0F2HwhezE2IizrqAz6pzLf9pIRYGteLsZlMjakCNpK6MooOwZtXwPpv4MrXodOIk3/OWq6kRPlk6VYe+3QVm/fkM6RTEp1b1D8qAeTkHaI4SDU3LjriqA/5sgmg9LGNCTCmelmCqKyDufDyBbBjBVz7EbQp2zs3PBUUFvPK/I08PXMt+YXFv/iQTyzzDT/wcb2YSGvXN6aGsgRxIvJ2wgvD4EAO3Pg5ND216p67lisuUSIE+9A3JgSUlyCsEfdY6iXCNe854yJeuwT2bvY7ohojMkIsORgTBixBlKdRijN4rmAfvHYp5O/2OyJjjKk2liCOp0V3uOoN2LUO3rgSCvP9jsgYY6qFJYiKSB0El0yEzO9g6o1QHLxfvjHGhBJLEBV12kUw8u+wejpMu8uZ5M8YY0JYhRKEiFwuIgnu/QdE5D0ROdPb0GqgXuNg4N3w/Svw9f/6HY0xxniqolcQf1TV/SIyAPg18DLwnHdh1WDnPABnXAOzH4eFE/2OxhhjPFPRBFHs/jwPeE5VPwRivAmphhOB85+AjiNg+j2w/AO/IzLGGE9UNEFsFpH/AKOB6SISW4lzQ09klDMteOte8N442DDH74iMMabKVfRDfjTOzKvDVXUP0Bi4x7OoaoOYunDVFGjcDqaMgW1L/Y7IGGOqVEUTRAtgmqquEZEhwOXAQs+iqi3qNoax70JsgjOQbneG3xEZY0yVqWiCeBcoFpFTgBeAVOANz6KqTRokO0mi6KAzJUfeTr8jMsaYKlHRBFGiqkXAJcATqnoXzlWFAWjaGca8DXuz4PXLndlgjTGmlqtogigUkauAa4FP3G3R3oRUS7XpDZe9CFuXwNvXQnHwBXGMMaa2qGiCuAHoCzyiqhtEJBV4zbuwaqlTRzpdYNd9BR/e7qxQZ4wxtVSFEoSqrgDuBpaKSFcgS1UfPd55IjJcRFaLyFoRufcYx4wWkRUislxE3nC39RCR+e62n0Tkikq8Jn/1vA7OfgB+mgJfPuh3NMYYc8KiKnKQ23PpZWAjIEBrEblOVWeXc04k8AwwFMgCFonIR26yKT2mA3Af0F9Vd4tIU3fXAeBat9dUS2CxiHzudrGt+QbdDbnbYd5TEN8M+t3ud0TGGFNpFUoQwP8Bw1R1NYCIdATeBHqWc04vYK2qrnfPmQKMAlYEHDMOeEZVdwOo6g7358+lB6jqFhHZASQBtSNBiMCIxyBvB8y4H+KbQvfRfkdljDGVUtEaRHRpcoDDH+DHK1K3AjIDHme52wJ1BDqKyFwRWSAiw8s+iYj0wpnWY12QfeNFJF1E0rOzsyv4UqpJRCRcPAFSBsIHt8Lar/yOyBhjKqWiCSJdRF4QkSHubSKw+DjnBFuTsuwc2VFAB2AIcBUwSUQaHn4CkRbAq8ANqnpUxVdVJ6hqmqqmJSUlVfClVKPoOLjydUjqDG9dA5u/9zsiY4ypsIomiFuB5cBvgTtwmoluOc45WUDrgMfJwJYgx3yoqoWqugFYjZMwEJH6wDTgAVVdUME4a564BjB2KtRr4oyRyDnqQsgYY2qkivZiOqiq/1TVS1T1YlX9l6oePM5pi4AOIpIqIjHAlcBHZY75ADgbQEQScZqc1rvHvw+8oqrvVOYF1UgJzWHs+4DCqxfD/u1+R2SMMcdVboIQkaVuN9Ogt/LOdUde344zyd9K4G1VXS4iD4vIhe5hnwM5IrIC+Bq4R1VzcCYHHARcLyJL3FuPk3yt/ko8Ba5+x5mK4/VLoWCf3xEZY0y5RMtZOlNE2pZ3sqrWmNnp0tLSND093e8wjm/tl/DGFdC2H1w9FaJi/Y7IGBPGRGSxqqYF21fuFYSqZpR38ybcEHfKuTDqWdgwG94bDyXFxz/HGGN8UNE1qfeLyL4yt0wReV9E2nkdZMg5/QoY9ldY8QF8di+UcxVnjDF+qehAuX/i9EB6A6f76pVAc5xeR5Nxuqmayuj3X7B/G8x/2hltPehuvyMyxphfqGiCGK6qvQMeTxCRBar6sIj8wYvAwsLQv0DuDpj5FydJnHmN3xEZY8xhFV4Pwp1UL8K9Bc4bYe0jJyoiAkY9A+3PgY/vgNWf+R2RMcYcVtEEcTVwDbDDvV0DjBWROjhdWc2JioqB0a9Ci9Phneth03d+R2SMMUDFB8qtV9ULVDXRvV2gqmtVNV9Vv/U6yJAXG++MkajfEt4YDTtW+R2RMcZUuBdTsttjaYeIbBeRd0Uk2evgwkq9RLjmPWdcxGuXwN7NfkdkjAlzFW1iehFnmoyWODOyfuxuM1WpUYozeK5gH7x2KeTv9jsiY0wYq2iCSFLVF1W1yL29hLM+g6lqLbrDVW/ArnXw5lVQmO93RMaYMFXRBLFTRMaKSKR7GwvkeBlYWEsdBJdMhE0LYOqNUFzkd0TGmDBU0QRxI84EetuArcBlwA1eBWWA0y6CkX+H1dNh2l022toYU+0qNFBOVTcBFwZuE5E7gSe8CMq4eo1zRlvP+QfEN4dz7vc7ImNMGKnoFUQwv6uyKMyxnfMAnHENzH4cFk3yOxpjTBip6FQbwQRbUtRUNRE4/wlnHYlpd0PdRKf5yRhjPHYyVxDWKF5dIqPgssnQuhe8Nw42zPE7ImNMGDjeinLBpvneJyL7ccZEmOoSUxeumgKN28GUMbBtqd8RGWNC3PEWDEpQ1fpBbgmqetzmKREZLiKrRWStiNx7jGNGi8gKEVkuIm8EbL9ORNa4t+sq/9JCUN3GMPZdiE1wBtLttjWbjDHeOZkmpnKJSCTwDDAC6AJcJSJdyhzTAbgP6K+qpwF3utsbAw8CvYFewIMi0sirWGuVBslOkig66EzJkbfT74iMMSHKswSB88G+1p3o7xAwBRhV5phxwDOquhtAVXe4238NfKGqu9x9XwDDPYy1dmnaGca8DXuznMn9Dub6HZExJgR5mSBaAZkBj7PcbYE6Ah1FZK6ILBCR4ZU4FxEZLyLpIpKenZ1dhaHXAm16w2UvwpYf4O1robjQ74iMMSHGyxs6WzEAABYGSURBVAQRrBts2Z5PUUAHnCVLrwImiUjDCp6Lqk5Q1TRVTUtKCsOpoU4d6XSBXfcVfHg7lJT4HZExJoSczDiI48kCWgc8TsZZ17rsMQtUtRDYICKrcRJGFr9c5zoZmOVZpLVZz+ucZUu//ivEN4Vhf/E7ImNMiPDyCmIR0EFEUkUkBrgSZ8rwQB8AZwOISCJOk9N64HNgmIg0covTw9xtJphBd8NZ42DeUzDvab+jMcaECM+uIFS1SERux/lgjwQmq+pyEXkYSFfVjziSCFYAxcA9qpoDICJ/wUkyAA+r6i6vYq31RGDEY5C3A2bc71xJdB99/POMMaYcoiEyS2haWpqmp6f7HYa/Cgvg9ctg03ynl9Mpv/I7ImNMDScii1U1Ldg+L5uYTHWLjoMrX4ekzvDWNbD5e78jMsbUYpYgQk1cAxg7Feo1gdcvh5x1fkdkjKmlLEGEooTmMPZ9QOHVi2H/dr8jMsbUQpYgQlXiKXD1O85UHK9fCgX7/I7IGFPLWIIIZa16whWvwI6V8NbVzvxNxhhTQZYgQt0p58KoZ2HDbHj/ZhttbYypMC9HUpua4vQr3DESD0C9JBjxuDN2whhjymEJIlz0+y/Yvw3mPw3xzZzR18YYUw5LEOFk6F+ceZtm/sVJEmde43dExpgazBJEOImIgFHPwIGd8PEdTnNTJ1tmwxgTnBWpw01UDIx+FVqcDu9cD5kL/Y7IGFNDWYIIR7HxzhiJ+i2d0dY7VvkdkTGmBrIEEa7qJcI170FUrLO29d7NfkdkjKlhLEGEs0YpcPVUZ5T1a5dC/m6/IzLGVJaqZzMlWIIIdy26w1VvwK518OZVUJjvd0TGmIooKYYVH8LEc+Cd6zz5FZYgDKQOgksmwKYFMPVGKC7yOyJjzLEUHYTFL8EzveDta50r/84XOlcSVcy6uRrHaRc7E/tNvxum3QUXPGWjrY2pSQr2QfpkWPAc5G5zeiJe/pKTHCIiPfmVliDMEb3GOaOt5/wD4pvDOff7HZExZv92+O45WPQCHNwH7YbAxc87Pz3+EudpghCR4cCTOGtST1LVR8vsvx74O1DaheZpVZ3k7nscOA+nGewL4A4NlfVRa7JzHoDc7TD7cUhoBmfd5HdExoSnnHUw7ylY8iaUFDpXCgPuhJZnVFsIniUIEYkEngGGAlnAIhH5SFVXlDn0LVW9vcy5/YD+QHd307fAYGCWV/Ealwic/4TT3DTtbme0dZdRfkdlTPjY8gN8+wSs/AgioqHHGGcutSbtqz0UL68gegFrVXU9gIhMAUYBZRNEMArEATGAANGALYtWXSKj4LLJ8OpF8O5NUKcxpA70OypjQpcqrJ8F3/4LNnwDsfWh/x3Q+1bnSt4nXvZiagVkBjzOcreVdamI/CQiU0WkNYCqzge+Bra6t89VdWXZE0VkvIiki0h6dnZ21b+CcBZTF66aAo3bwZQxsG2p3xEZE3pKimHZezBhsPOFLHs1DH0Y7loO5/7Z1+QA3iaIYNWTsjWEj4EUVe0OfAm8DCAipwCdgWScpHKOiAw66slUJ6hqmqqmJSUlVWnwBqjbGMa+C7EJzkC63Rl+R2RMaCgscHokPZ0GU2+AQ3lOz8E7f3KuHOLq+x0h4G2CyAJaBzxOBrYEHqCqOapaug7mRKCne/9iYIGq5qpqLvAp0MfDWM2xNEh2kkTRQWdKjrydfkdkTO2Vvwfm/B880Q0+uQviGjqTZ/6/hdDzOmfqmxrEywSxCOggIqkiEgNcCXwUeICItAh4eCFQ2oy0CRgsIlEiEo1ToD6qiclUk6adYczbsDcL3hgNB3P9jsiY2mXfVpjxR/hXV/jqYWjeDa77GMbNhC7ejWM4WZ4VqVW1SERuBz7H6eY6WVWXi8jDQLqqfgT8VkQuBIqAXcD17ulTgXOApTjNUp+p6sdexWoqoE1vuOxFeOtqZ/TmmLcgMtrvqIyp2XauhXlPwo9ToKTIGZDa/w5nkFstIKEytCAtLU3T09P9DiP0LX4ZPv4ttOgBv34EUgb4HZExNU/WYpj7L1j5idNsdMZY6Hs7NE71O7KjiMhiVU0Lts9GUpvK6Xmds57EjD/BS+dBp5FOr4vEDn5HZoy/VGHdV84Yho1zIK4BDPw99L4F4mtnJxpLEKbyul7qJIYFz8Gcf8IzvSHtRhhyr7POhDHhpLgIVnwAc59wuoMntIRhf4We1zs9AGsxa2IyJyc3G755FNJfhOi6MPB30OdWiK7jd2TGeKswH354Deb9G/ZkQJMOzlQY3UY7S/vWEuU1MVmCMFUj+2f48kFYPR0atIZf/Qm6XgYRNqO8CTH5u2HRJFjwPBzYCclnQf87navqWvj/3RKEqT4b5sCM+2Hrj1bINqFl72ZY8KyzFsOhXDhlKAy4C9r2q9VT41uR2lSf1IEwbhYsfcfp722FbFPbZa+GuU/BT2+Bljg1uP53QPOufkfmOUsQpupFRMDpVzgDgKyQbWqrzEVO4XnVJxBVB9JucLqqNmrrd2TVxpqYjPdys+Gbx5y5Z6yQbWoyVVjzhZMYMuY6U2H0vhl6jQ/ZLzZWgzA1Q2Ahu36yU8judnmtLOyZEFNcBMvfg7lPwvZlzv/Pvv8PzrzWGfcTwixBmJrFCtmmpjh04EhX1b2bIOlUp0dSt8vCZioZK1KbmsUK2cZvB3bBwomw8D9wIAda94aRj0OHX9sVbQBLEMYfVsg2ftibBfOfceYUK8yDjsOdK4a2ff2OrEayBGH8FV3HKVqfcc2RQvaPU6yQbarWjpVOfWHpO87jrpc5XVWbdfE3rhrOahCmZrFCtqlKmxY4k+f9/KnTg+7M65zic8PWxz83TFiR2tQ+ZQvZw/7q1C6MOZ6SElgzw+mqumk+1Gl8pKtq3cZ+R1fjWJHa1D5lC9kvn+8Uss99CJI6+h2dqYmKC2HpVKcpKXulMyfYiMedtRhi6vkdXa1kCcLUXMEK2c/2sUK2+aVDefD9K07xeW8mNO0CF0+ArpeETVdVr1gTk6k9bES2CZSXAwsnOF1V83dDm37O5HkdhtbqyfOqW3lNTJ5W/kRkuIisFpG1InJvkP3Xi0i2iCxxbzcF7GsjIjNEZKWIrBCRFC9jNbVAfBKc9w+4bYHTBPXVQ/DvNPjxLafd2YSHPZtg+n/Dv05z1iJp0xdunAE3fgodh1lyqEKeXUGISCTwMzAUyAIWAVep6oqAY64H0lT19iDnzwIeUdUvRCQeKFHVA8f6fXYFEYaskB1eti93u6pOdZJA9yug32+h6al+R1ar+VWk7gWsVdX1bhBTgFHAinLPco7tAkSp6hcAqprrYZymtrJCduhTdXoiffsvp2dSdD1njee+t0GDZL+jC3leJohWQGbA4yygd5DjLhWRQThXG3epaibQEdgjIu8BqcCXwL2qWhx4ooiMB8YDtGnTpupfgan5jlnIvgGG3GeF7NqqpMQZu/DtE5C1EOo2gbMfgLN+Y11Vq5GXNYhgDYFl27M+BlJUtTtOEnjZ3R4FDATuBs4C2gHXH/VkqhNUNU1V05KSkqoqblMblY7I/u0PTi+n9BfhyR5OwijM9zs6UxElJbB9BXz3HyfJTxkDudtg5D/gzmUw+B5LDtXMyyuILCBwuGIysCXwAFXNCXg4EXgs4NwfApqnPgD6AC94Fq0JDaWF7F7jnRHZXz0Ei16wEdk1UUmxM7X2xrnO2gsZ8yB/l7OvWTe49AXochFEWm98v3j5zi8COohIKrAZuBIYE3iAiLRQ1a3uwwuBlQHnNhKRJFXNBs4BrAJtKi6pI1z1plvIfgDeH++sJ2yFbP8UFzkdCjK+dZJBxnw4uNfZ17AtdBrhrO/ctj80SrHeSDWAZwlCVYtE5HbgcyASmKyqy0XkYSBdVT8CfisiFwJFwC7cZiRVLRaRu4GvRESAxThXGMZUTupAGPe1FbL9UHQItnzvXB1snAuZ38Eht79Jk1PgtFHQdgCk9LeCcw1lA+VM+CjMP1LILjzgFLIH3+s0S5mTV1gAm9PdJqNvnTWdi9z6T1Jn5+ogpb9zhZDQ3N9YzWE2WZ8xgY4akX0X9LnNRmRX1qE8yFx45AphczoUHwIEmnU9kgza9rPeZDWYJQhjgrGpxSunYJ/TTLTxWycpbPkBSopAIqDF6U4ySBkAbfpAnUZ+R2sqyBKEMeUpLWRvXWIjsgPl73YKyRluL6OtP4KWQEQUtDzTvUIYAK17QVx9v6M1J8gShDHHU1ICy6bClw/BvqzwLGTn7TzS3XTjXKcLKgqRsZCcdqS5qHUvmz47hFiCMKaiwqmQvX/bkfpBxlzIXuVsj6rjJIGUAU5CaJUG0XH+xmo8YwnCmMoKxUL23qwjPYw2zoVd65ztMfHQuveRJqOWZ0BUjL+xmmpjCcKYE1VbC9mqsHtjwBXCt8402QCxDaBtX7eo3B+an26jlcOYJQhjTtYvCtmnw7BHalYhWxVy1h7pYZQxD/ZtdvbVaeyOQRjgJIVmp0FEpL/xmhrD1qQ25mSVjsguLWT7PSK7pMSpGWTMdZPCPMjb4eyr1/TIGISUAZDYqeZf8ZgayRKEMRUVEQHdR0PnC46eWtzrQnbpxHYZ844khNKJ7eq3gnZDjiSFJqfYPEamSlgTkzEnKm8nzHrUm0J2cRFs+zFgptMyE9uV9jCyie3MSbIahDFeqopCdtEhZ2RyaQ+jshPbte1/pKhsE9uZKmQJwpjqUJlCtk1sZ2oISxDGVJeyI7I7joChD0ODVjaxnamRLEEYU93KjsgWsYntTI1k3VyNqW6la2SfeS3MfwZQm9jO1DqWIIzxUr1EOPdBv6Mw5oR4OnpGRIaLyGoRWSsi9wbZf72IZIvIEvd2U5n99UVks4g87WWcxhhjjubZFYSIRALPAEOBLGCRiHykqivKHPqWqt5+jKf5C/CNVzEaY4w5Ni+vIHoBa1V1vaoeAqYAoyp6soj0BJoBMzyKzxhjTDm8TBCtgMyAx1nutrIuFZGfRGSqiLQGEJEI4P+AezyMzxhjTDm8TBDBxv6X7VP7MZCiqt2BL4GX3e23AdNVNZNyiMh4EUkXkfTs7OyTDtgYY8wRXvZiygJaBzxOBrYEHqCqOQEPJwKPuff7AgNF5DYgHogRkVxVvbfM+ROACeCMg6ja8I0xJrx5mSAWAR1EJBXYDFwJjAk8QERaqOpW9+GFwEoAVb064JjrgbSyycEYY4y3PEsQqlokIrcDnwORwGRVXS4iDwPpqvoR8FsRuRAoAnYB13sVjzHGmMoJmak2RCQbyDiJp0gEdlZROFXJ4qoci6tyLK7KCcW42qpq0MVMQiZBnCwRST/WfCR+srgqx+KqHIurcsItLluH0BhjTFCWIIwxxgRlCeKICX4HcAwWV+VYXJVjcVVOWMVlNQhjjDFB2RWEMcaYoCxBGGOMCSpsE4SIXC4iy0WkRESO2T3seGtaeBBXYxH5QkTWuD+DrkcpIsUB62h85GE8x1vTI1ZE3nL3fyciKV7FUomYyl1nxMO4JovIDhFZdoz9IiJPuXH/JCJn1pC4hojI3oD360/VFFdrEflaRFa6f4t3BDmm2t+zCsZV7e+ZiMSJyEIR+dGN66Egx1Tt36OqhuUN6Ax0AmbhTOUR7JhIYB3QDogBfgS6eBzX48C97v17gceOcVxuNbxHx339OBMrPu/evxJnfQ+/Y7oeeNqH/1ODgDOBZcfYPxL4FGciyz7AdzUkriHAJz68Xy2AM937CcDPQf4tq/09q2Bc1f6eue9BvHs/GvgO6FPmmCr9ewzbKwhVXamqq49z2EmtaXGCRnFkVtuXgYs8/n3lqcjrD4x3KvArEQk2k291xuQLVZ2NM2XMsYwCXlHHAqChiLSoAXH5QlW3qur37v39OHOxlV0SoNrfswrGVe3c9yDXfRjt3sr2MqrSv8ewTRAVVNE1LapSM3UnMHR/Nj3GcXHuVOcLRMSrJFKR13/4GFUtAvYCTTyKp6IxQZB1RmoAP/4/VVRft+niUxE5rbp/udsUcgbOt+JAvr5n5cQFPrxnIhIpIkuAHcAXqnrM96sq/h69nM3VdyLyJdA8yK77VfXDijxFkG0n3S+4vLgq8TRtVHWLiLQDZorIUlVdd7KxlVGR1+/Je1SOiq4z8qaqHhSRW3C+UZ3jYUwVVd3vVUV9jzMfT66IjAQ+ADpU1y8XkXjgXeBOVd1XdneQU6rlPTtOXL68Z6paDPQQkYbA+yLSVVUDa0tV+n6FdIJQ1XNP8imOu6bFiSgvLhHZLu406O6l9I5jPMcW9+d6EZmF8y2nqhNERV5/6TFZIhIFNMDb5oyTWWfEb578fzpZgR9+qjpdRJ4VkURV9XxSOhGJxvkQfl1V3wtyiC/v2fHi8vM9c3/nHvfvfjgQmCCq9O/RmpjKd3hNCxGJwSn6eNZjyPURcJ17/zrgqCsdEWkkIrHu/USgP7DCg1gq8voD470MmKluhcwjx42pTBv14XVGaoCPgGvdnjl9gL16ZD0U34hI89J2ahHphfO5kFP+WVXyewV4AVipqv88xmHV/p5VJC4/3jMRSXKvHBCROsC5wKoyh1Xt32N1VuFr0g24GCfbHgS2A5+721viLHdaetxInF4M63CapryOqwnwFbDG/dnY3Z4GTHLv9wOW4vTgWQr8xsN4jnr9wMPAhe79OOAdYC2wEGhXDe/R8WL6G7DcfX++Bk6tpv9TbwJbgUL3/9ZvgFuAW9z9Ajzjxr2UY/Se8yGu2wPerwVAv2qKawBO88dPwBL3NtLv96yCcVX7ewZ0B35w41oG/Mnd7tnfo021YYwxJihrYjLGGBOUJQhjjDFBWYIwxhgTlCUIY4wxQVmCMMYYE5QlCGMqQURyj39UuedPdUe/IyLxIvIfEVnnzs45W0R6i0iMez+kB7Kams8ShDHVxJ2vJ1JV17ubJuGMcu2gqqfhzEKbqM4khF8BV/gSqDEuSxDGnAB3ZO/fRWSZiCwVkSvc7RHutAvLReQTEZkuIpe5p12NOzJeRNoDvYEHVLUEnGlTVHWae+wH7vHG+MYuYY05MZcAPYDTgURgkYjMxpn2JAXohjMT70pgsntOf5xRzQCnAUvUmXwtmGXAWZ5EbkwF2RWEMSdmAM6MscWquh34BucDfQDwjqqWqOo2nKk+SrUAsivy5G7iOCQiCVUctzEVZgnCmBNzrEVYylucJR9nrhxw5vE5XUTK+xuMBQpOIDZjqoQlCGNOzGzgCncBlyScZT0XAt/iLFYUISLNcJamLLUSOAVAnbU70oGHAmYF7SAio9z7TYBsVS2srhdkTFmWIIw5Me/jzKr5IzAT+G+3SeldnBlTlwH/wVmJbK97zjR+mTBuwlk4aq2ILMVZu6J0rYOzgenevgRjymezuRpTxUQkXp2VxprgXFX0V9Vt7hz+X7uPj1WcLn2O94D79PjrphvjGevFZEzV+8Rd2CUG+It7ZYGq5ovIgzjrBm861snuQkgfWHIwfrMrCGOMMUFZDcIYY0xQliCMMcYEZQnCGGNMUJYgjDHGBGUJwhhjTFD/H5txkGGa9vz+AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "### 最佳超参数\n",
    "print(\"Best score: %f using %s\" % (-grid.best_score_, grid.best_params_))\n",
    "\n",
    "# plot CV误差曲线\n",
    "test_means = -grid.cv_results_[ 'mean_test_score' ]\n",
    "train_means = -grid.cv_results_[ 'mean_train_score' ]\n",
    "\n",
    "print(test_means)\n",
    "print(train_means)\n",
    "\n",
    "plt.plot(np.log10(C_s), test_means,label='test')\n",
    "plt.plot(np.log10(C_s), train_means,label='train')\n",
    "\n",
    "plt.legend()\n",
    "plt.xlabel( 'log(C)' )                                                                                                      \n",
    "plt.ylabel( 'Logloss' )\n",
    "\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "参数C：惩罚系数，即对误差的宽容度，是为了在模型准确率和复杂度之间取得一个平衡。C越高，说明越不能容忍出现误差，容易过拟合；C越小，容易欠拟合。C过大或者是过小，泛化能力都会变差\n",
    "\n",
    "随着C的增大，模型越来越复杂，训练误差和交叉验证得到的测试误差都是先减小后增加，最佳模型的选择是根据测试误差决定的\n",
    "\n",
    "从这里的训练误差结果可以看出，通过概率估计方法的结果并不能很好地评估模型，因为logloss在训练集上应该是随着C的增大，即模型复杂度的增大，单调递减的"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### RBF核SVM正则参数调优\n",
    "\n",
    "RBF核是SVM最常用的核函数。\n",
    "RBF核SVM 的需要调整正则超参数包括C（正则系数，一般在log域（取log后的值）均匀设置候选参数）和核函数的宽度gamma\n",
    "C越小，决策边界越平滑； \n",
    "gamma越小，决策边界越平滑。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=3, error_score=nan,\n",
       "             estimator=SVC(C=1.0, break_ties=False, cache_size=200,\n",
       "                           class_weight=None, coef0=0.0,\n",
       "                           decision_function_shape='ovr', degree=3,\n",
       "                           gamma='scale', kernel='rbf', max_iter=-1,\n",
       "                           probability=True, random_state=None, shrinking=True,\n",
       "                           tol=0.001, verbose=False),\n",
       "             iid='deprecated', n_jobs=4,\n",
       "             param_grid={'C': array([1.e-01, 1.e+00, 1.e+01, 1.e+02, 1.e+03]),\n",
       "                         'gamma': array([ 0.1,  1. , 10. ])},\n",
       "             pre_dispatch='2*n_jobs', refit=True, return_train_score='warn',\n",
       "             scoring='neg_log_loss', verbose=0)"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.svm import SVC\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "\n",
    "Cs = np.logspace(-1, 3, 5)# logspace(a,b,N)把10的a次方到10的b次方区间分成N份 \n",
    "gammas = np.logspace(-1, 1, 3) \n",
    "param_rbf = {'C': Cs, 'gamma': gammas}\n",
    "\n",
    "rbf_grid = GridSearchCV(SVC(kernel='rbf',probability=True), param_rbf,cv=3, n_jobs = 4,return_train_score='warn',scoring='neg_log_loss')\n",
    "\n",
    "rbf_grid.fit(X_train_selected,y_train_selected)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best score: 0.634582 using {'C': 1.0}\n",
      "[[0.70867265 0.66705411 1.03602207]\n",
      " [0.65594928 0.59799856 0.87988501]\n",
      " [0.6233763  0.59156074 0.87289036]\n",
      " [0.61787752 0.61969834 0.88045438]\n",
      " [0.65495271 0.63337752 0.88086679]]\n",
      "[[0.67798337 0.59170072 0.40989947]\n",
      " [0.61194667 0.40779591 0.13237601]\n",
      " [0.51534569 0.22270671 0.06771382]\n",
      " [0.39056723 0.22392749 0.07699401]\n",
      " [0.35106334 0.24522946 0.07248802]]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nOydeXxU1dnHv+fOmmSy7wuQAIFICIICgopAEQFRkVcrglZra93rUnFrpS6vVmtbqlYpbtTlFa0biIpbWxCpIIIgBAQCJEhCyL5NkpnJzJz3jzuZTFYiZIOc7+czn7n3nnPPPDOB87vPec55jpBSolAoFIr+i9bbBigUCoWid1FCoFAoFP0cJQQKhULRz1FCoFAoFP0cJQQKhULRzzH2tgE/lpiYGJmamtrbZigUCsUJxZYtW0qllLFtlZ1wQpCamsrmzZt72wyFQqE4oRBCHGyvTA0NKRQKRT9HCYFCoVD0c5QQKBQKRT9HCYFCoVD0c5QQKBQKRT9HCYFCoVD0c5QQKBQKRT+n3whByQ81bF6dS121q7dNUSgUij7FCbeg7Fg5tLucr1fl8s1HeQw5LY6syckkDAlHCNHbpikUCkWv0m+E4LTzBjH41Fh2fJHP7g1HyPmmiJgBNrImp5A+Ph6T2dDbJioUCkWvIE60HcrGjh0rjzfFRIPTw95NR9ixtoCyAjuWYCMZExMZOTmZiLjgLrJUoVAo+g5CiC1SyrFtlvVHIWhESknhvip2fJHPgW9L8HolAzOjyJqcwsCR0WiaGjZSKBQnBx0JQbcNDQkhlgEXAMVSypFtlAvgKeB8oA74uZTy2+6ypx0bSUqPICk9gtoqJ7vWHyZ7XQEfLdlOWIyVzHOSGXFmElabqSfNUigUih6l2zwCIcQ5gB14tR0hOB/4NboQnAE8JaU842jtdqVH0BYej5fcbaXsWJvP4ZxKDCaN9LFxZE1JIW5QWLd9rkKhUHQnveIRSCnXCSFSO6gyB10kJLBRCBEhhEiUUhZ2l02dwWDQGHp6HENPj6OswM6OLwrY8/URdm84QnxaGFmTkxlyehxGkwouKxSKk4PenDWUDBwKOM/3XetVIQgkOtnGlAXDmTh3CHs2FrJjbQH/evl71r+zjxFnJZF5ThJh0UG9baZCoVAcF70pBG1FYtscpxJCXAdcBzBw4MDutKlNLEFGRk0dQNaUFPL3VLBjTT5bPzvI1s8OkjoqhqzJKaRkRCJUcFmhUJyA9KYQ5AMDAs5TgMNtVZRSPg88D3qMoPtNaxshBAMyohiQEUVNuYOd6wrY9d/D5H5XSkR8MCPPSSbjzEQsQf1meYZCoTgJ6M0eaxVwixDiTfRgcVVvxwd+DKFRViZcPIRxs9PY920xO9bms/7tHDauOsDw8fFkTUkhOtnW22YqFArFUenO6aNvAFOAGCFEPvAAYAKQUi4FVqPPGNqHPn30mu6ypTsxmDSGn5HA8DMSKPmhhh1r89m98Qg7vzxM4tBwsqakMHhMLAZDv0nrpFAoTjD69YKy7sJhb+D7rwrJXpdPdamD4HAzmWcnkTkpmZAIS2+bp1Ao+iFqZXEvIb2SgzvL2LG2gB92laEJQdroWEZNTSZxaIRKeKdQKHqMXllHoAChCVKzYkjNiqGqpI7sLwr4/qtC9n9bTFRSCFlTUhg2Ph6zVf0ZFApF76E8gh6mweUh55sidqzNp/SQHbPV4E94F5kQ0tvmKRSKkxTlEfQhTGYDI85K4pQzEynKrWbH2nyy1xWwfU0+KRmRZE1JITUrGk0FlxUKRQ+hPII+QF21i13rD7PzywLsFU5sURZGnpPMiLOSCAo197Z5CoXiJEAFi08QvB4vedvL2PFFPvm7K9CMgqGn6wnv4lPDVHBZoVAcM2po6ARBM2gMHhPL4DGxlBfWkv1FAbs3FrL36yJiB4aSNSWZ9LHxGNVuagqFogtRHkEfx+Vws/frI2xfW0BFYS2WECMjzkxi5ORkwmJUwjuFQtE51NDQSYCUksN7K/Xd1LaVIqVk0MhofTe1EVEq4Z1CoegQNTR0EiCEIHl4JMnDI7FXONj55WF2rj/Mh898R1hsEFmTk8mYmIg1RO2mplAofhzKIziB8bi9HNhawo61+RTur8Jo0kgfH0/W5BRiB4b2tnkKhaIPoTyCkxSDUSN9XDzp4+Ipza9hx9oC9m46wvf/LSRhcDhZU5IZclocBqNak6BQKNpHeQQnGc66BnZvOMKOL/KpKq4nKNTECF/Cu9Aoa2+bp1AoegkVLO6HSK/k0Pfl7PiigLwdpQghSDs1hqzJySQPj1RrEhSKfoYaGuqHCE0wMDOagZnRVJfWs/PLAnatL+TA1hIiE4LJmpLC8DMSMKvd1BSKfo/yCPoR7gYP+zbru6kVH6zBZDEwfEICWZNTiEpSCe8UipMZ5REoADCa9EynGRN9Ce++yOf7/xaS/UUBycMjyJqcQtqpMSrhnULRz1AeQT+n3u7yi0FNuYOQCAuZk5IYcXYSIeFqNzWF4mRBBYsVR8XrlRzcUcqOLwo4tKsczaDHGIJsJjSDQNMEmkHTj5u9tObHWhtlbV1rfGkt22x9LgQquH2cSCnxeiVej8Tr9urvHonH4zt2S7zeputejxePu+m4Wd1WdZpfb17P2+wzPS2ut7Sx/S/Q8vu0X9hhMy3LOqh8tK6xI3tbf067Jz/K3nGzU0kfG9+xYe2ghoYUR0XTBGmnxpJ2aiyVRfpuank7Sil1e33/eZv+c0uP3qn0qH2dFZc2Baul+LQhYO21qbUvUI3HBoNAaLoI6p2tF6+7RYfo1a/5O9CWHWU7nW97HavH3Uan3Kq9gE6/h/5emkGgGTUMLX9rY9NvZgj4/QL1XQgBzfS+ufh39CzQqqzFhQ6a7fhzREc1OzaqY3s7brg9ey3B3dNlK49AcUw0e8L0iUOrDssrW3VWjZ1S606wjXNvG+Xedup31GZ797RR3msIMLQhNgZjCxHSBAZjGyLWrJNt3fm27ICbXW+jPYOhpfg12tKyzaZjoQnlufVhlEeg6HKE0DsLw0mUEVtKiQwUjfbExdvG0IevvqbpT8RtdZStOtDAjlUlDVT0IkoIFAofQgiEQaCdROKmUHQGNU9QoVAo+jlKCBQKhaKfo4RAoVAo+jlKCBQKhaKfo4RAoVAo+jlKCBQKhaKfo4RAoVAo+jlKCBQKhaKfo4RAoVAo+jndKgRCiJlCiD1CiH1CiHvbKB8ohFgjhNgqhNguhDi/O+1RKBQKRWu6TQiEEAbgWWAWMAKYL4QY0aLa/cBbUsoxwOXAku6yR6FQKBRt050ewXhgn5TygJTSBbwJzGlRRwJhvuNw4HA32qNQKBSKNuhOIUgGDgWc5/uuBfIgcKUQIh9YDfy6rYaEENcJITYLITaXlJR0h60KhULRb+lOIWgrr27LhO/zgZellCnA+cBrQohWNkkpn5dSjpVSjo2Nje0GUxUKhaL/0p1CkA8MCDhPofXQzy+BtwCklBsAKxDTjTYpFAqFogXdKQTfAOlCiDQhhBk9GLyqRZ0fgGkAQohT0IVAjf0oFApFD9JtQiCldAO3AJ8C36PPDtophHhYCHGRr9qdwK+EEN8BbwA/lyfa3pkKhUJxgtOtO5RJKVejB4EDr/0+4HgXcFZ32qBQKBSKjlErixUKhaKfo4RAoVAo+jlKCBQKhaKfo4RAoVAo+jlKCBQKhaKfo4RAoVAo+jlKCBQKhaKfo4RAoVAo+jlKCBQKhaKfo4RAoVAo+jlKCBQKhaKfo4RAoVAo+jndmnSuL/H5riJWbisgNTqYQdEhpMWEMCg6mFibBSHa2kNHoVAo+gf9Rggqal3sLKjik+wjeLxNma6DzQYGRYeQGh1MakyIXyhSo0OIC7WgaUokFArFyU2/EYLLxg3gsnEDaPB4OVxZT15ZHXmlteSV1XKwrI49RTX86/siGjxNImE1aaRG656D/t4kGAlhViUSCoXipKDfCEEjJoPGIF+nPnlY8/2P3R4vhVUO8spqySur46BPKPaX1LJmdwkuj9df12zUGBTVOMzU5EUMig4mKSIIgxIJhUJxgtDvhKAjjAaNAVHBDIgKZlJ68zKPV3Kk2tHMi8gr1d+/zCnB6Q4QCYPGgKigJi/CJxRp0SEkRVgxGlSMXqFQ9B2UEHQSgyZIjggiOSKIs4bGNCvzeiVFNQ7ySus46PMmGgXjq/1l1Dd4/HWNmmBAVLB/uCk1OphBMbo3kRIZhEmJhEKh6GH6lxAc+gaEgNgMsNi6rFlNEySGB5EYHsTEIdHNyqSUlNQ4W8Uk8spq+Sa3nFpXk0g0ik3zoLUek0iJDMJiNHSZzQqFQtFI/xKC/zwMuev048hUiMuEQWfCmbfo16TUhaILEUIQF2YlLszK+LSoZmVSSkrtLr8XEehNbD1YQY3T7a+rCUiKCGoWvG4UjAFRwVhNSiQUCsWxIaSUR6/Vhxg7dqzcvHnzj76vpK6EmtI9DKwtx1i8G4p26q/IVLjyHb3S0kkgNIjPhLgRED8C4rPAFtth292BlJKKugY9cF3aXCgOltVSWdfgrysEJIZZffGIAG8iJphBUSEEmZVIKBT9HSHEFinl2LbK+o1H8MGBD/jrlr9iMVgYGjGUjJQMhmXNICNyGMNcdmymEBgyFQq3Q87nsO11/cbRV8DFS3Rv4fPfQ+xwXSRiM8Ac3G32CiGICjETFWLmtIGRrcor61xN4uCLTeSW1fLpziOU17qa1U0IszZNgY0JJs0XxB4UHUyIpd/8E1AoFO3QbzyC/Jp8vi3+lt3lu9lbvpfdFbupclb5y1NsKQyPGs7wyOH6uzWOpJpSRFAEJI2GmiJ46lRw1/vuEBA9BCbfC6N+Cm4nVOVDZBpovRvwrapv4IeyOnLLan1TYJu8iVK7s1nd2FCLHofweRNNayaCCbWaeukbKBSKrqYjj6DfCEFLpJQU1RWxt2Ivu8t3s6d8D3sr9nKw+iAS/TcJNYUyLGoYGVEZukBEpDNEGrCU5ujDSsU74bSrIX06HNoEL00HUzDEneIbWsqEUy6C8OTjtrerqHE0cLCszh+wbvQo8spqKa5pLhKRwSZiQy3E2CxE2yzE2Mz6cYjv3XceY7Oo4SeFogto8HgptTsprnZSVO2guMZJse+9qNrBVWemMnV43DG1rYaG2kAIQUJIAgkhCZyTco7/el1DHTmVOewp38Oe8j3srtjNeznvUe/zBAzCQFp4mu41ZE5jeEgIw+vLiI5MhYueaRKIPath62uQeKouBDmfw8YlPoEYqccfYoaDydqj3zvUamJkcjgjk8NbldU63fxQ7htmKq3jUEUdpTVOympd7MivpNTuwh4QwA4k2Gzwi0N0iIXYUP09xmYm2nc91icoEUEmtSpb0a9o8HgpqWm7c9ffnZTUOCirddHy2VwIiLFZiAu14AiYZdiV9FuP4MfglV4O1Rzyew57KnSRKKor8teJDYplWNQwhkcO1z2IiGEMMgRhCI4GowW+/wDW/QmKd4PH9+QtDHD7dghPgfwtYC/SBSJ8YK8PL7WHo8FDWa3LJxBOSu0uSu1Oylq8l9pdlNc68bbxz8ug6fGP6BAzsaG6hxFts/iFJLZRUHzeh5oRpeirON0eSmqcvs7dSXGNo9nTfFG1gxLfw1RLNF8HHx9mJS7Uos8uDA0814+jQ8xdsghVDQ11E5WOSr8oNL7vr9qP26s/NVsNVoZGDNW9B1/8YVjYYGz2Yt1rKP5ejzFoGrx/i+5BAJhtuueQMBJmL9YfCdwuMJp78dv+eLxeSWV9g08YdHEoayEWupjo1+raedoJtRj9w1BN782HqqJtFmJtFsKCjCqbrOK4cbo9vo69rad3h//pviJg9l4jBk0QYzO328E3vkfbLD2aikYJQQ/S4GngQNUB9lTs6TAwnRGV0cyDSDTaECV7dIEo2glFu/TA9K/+o9/0f5fowuGf1joSEkZBXEYvfdOup87lbi4Sdv1JqvGJqszeJBrlda1daACTQRAd0uRRxPjjGGb/9ca4RlSIGbOxb3peiu7B0eBpNkTTNFQT8DRf42g2PbsRoyaIDbW027nH+t6jQszH1MG7vW6cHicOtwOHx4HT7cThcfjPHW4HwyKHkRKackzfXQlBL9MYmA70HPZU7OGH6h+aAtPm0KYZS773oRFDMRt8XsDmf8APG3SBKNkN3gYYdDZc85Fe/vnvwRquC0TcCH246SR+MnZ7vFTUNegeRY2rxTBVc6+j1O5slgsqkPAgU4A4NMY1LM2v+d5tFuVt9FUcDR5/J95eoLW4xklVfesO3mQQxNraGJoJtRAVaiAiGMKCIcjsxeVt3jm37KwDO3KHO+Dcd1zvrsfpcTY7bqzfOJLQEfefcT/zMuYd02+khKCPUtdQx96KvU0zlyr2kFOR4w9MG4WR1PBUhkcNJyOyyYOINodB2T5wOyBpDHi98MxYKN/f1LglHCbeBFPu1ddAHNqkew/W1kHikx0pJbUuj9+j0L2NJtHwC0it/t7W0yDoGWdjQszEhAbOnGryOiJDzJgMAqOmYdAEBk1gbPHedF1rcd78XQmOTr3L0+zpvajawZGaGo5U2ym2V1Nsr6WstobaBgdCawDR4H83Gt3YrJKQIEmw2YvV4sFs8mAyejAYGtA0N1K48MgGnJ7WHbvT48Qjjy04a9bMWI1WrAYrVqMVi9HSdGxofhxkDMJisGAxWggyBLWu62vHYrSQHJJMhDXimGw6biEQQvwU+ERKWSOEuB84DXhESvntMVl0HJxMQtAWHq+HQzWHWsUe2gpMZ0Rm+OMPg0IHYXDZ9eGjomzdcxg4AUZdBtWFsNg3hBQ+UB9aihsBmXMhcVQvfdO+S4PHS0WtixKfZ9HodZQGeB+BwfHA9ORdhSZoVyxai4eG1kJMWp4bNA2D1tRmR3UMmtbBZzVe66COQaAJn10CGqiltqGaek81dZ4a6txV1LqrsTdUU+mopcpRR42zDntDPXUN9Tjc+hOz2+vEiwuadfBuhPjxD68C0apjbatTDux02+qsO1NuMVjQRN8bcuwKIdgupRwlhDgbeAz4M/BbKeUZXWvq0TnZhaA9GgPTu8t3+z2IA5UHcMumwHR6ZDrDIofpHkRUBsMihxFiCgFXHeR92SQQxbugdC/MeRZOvRwKv4OVNzcJROP01tDEk3p4qSuQUlLjdFNa46SizkWDR+L1Stxeicf/7sXjBbfX67/WXh2P19vsevN6nanTeO4NaLvpFdiWt4VNLesE7uSn40EY6hGGOt+r1n+MoQ5hrA0oayyvR4i2hVJKAV4zUprAa0RIM0bNjFmzYDFaCTJaCTEFYTMHEWYNJsIaQmSQ/t7WU3arzrrx2GjFrJn7vZfVFUKwVUo5RgjxGLBDSrm88dpR7psJPAUYgBellI+3Uecy4EFAAt9JKRd01GZ/FYK2aAxMNw4rNXoQgYHpAaEDWsUeEkMSEZ4GkF59HUPBt7DmD7pAVBc0fcDVH0DaOVB9GFy1ED1UCcNJgsvjotJZqb8clU3HvleVs4oKR0WzOjUNNe22Z9LM2Exh2EzhhBrDCTGFEWIMI9gYRrAhjCBjGEGGUKwiFKsWikULw2wIJibESnyYhbhQq5rx1c10xYKyAiHEc8C5wB+FEBagQ99HCGEAngWmA/nAN0KIVVLKXQF10oH7gLOklBVCiGNbMtdPMRlM/qGhRloGphs9iH//8O9WgelGryEjKoMh85frgem6ct/w0k5IyNIb3fIKfPG47iGknQNpk/X3iAG98bUVAUgpqXfXN+/IHQGdubOiWcde5ayi0llJnbuu3TaDjEFEWiIJt4QTYYkgJTSFCEtE65e16TjIGKQ68ROYznoEwcBMdG8gRwiRCGRJKT/r4J6JwINSyhm+8/sApJSPBdR5AtgrpXyxswYrj+DY6Gxg2p9OI2o4g8MHYzVaMVQXYsz7EkPueox56xF1pXoqjXsO6msbjuwAW0KvZGk9mZBSUtNQ0/oJ3dHG07qzgiqH3qm7vK0XKzUSag4lwhLRrGMP7MAbX+GWcCKtkURYIppmqilOKrpiaGgIkC+ldAohpgCjgFellJUd3HMpMFNKea3v/GfAGVLKWwLqrAT2AmehDx89KKX8pI22rgOuAxg4cODpBw8ePKrNiqPTVmB6d/luiuuKO7zPIDQMaBgNZgyaAZPTjsHTgFEzYTAGYTQHYzSHYjCYMWpGDJoBozDqx8Lgv2bSTBiEoVl5yzot72uvvPF6YLsd3WcSJv3+9uoIw3E94bq9bqpd1Z1+Qm+81t4sFU1o/g67vSfzcEs4kZZI/3G4JRyj1m+zyCha0BVDQ+8CY4UQQ4GXgFXAcuD8jj63jWstVccIpANTgBTgSyHEyJYCI6V8HngedI+gkza3onLlSgwREdgmT1ZuLGDQDKSGp5IansqM1Bn+6xWOCn8CPpfHhUd6aPA24PF68EgPbq8bt3Tj9rr1a/YjuKvycdccxl1TikeW4A5Lwp08Bo/Xg7uuFI/JhNvrxiEd+jWvu6ktX3uN7Td+VmOdY53Cd9y/TxtiYhS6kDS71igcCH/nX+2qbrddk2Zq1oEPiRjS5pN5Y8cebgkn1BzaJ2eiKE4OOisEXimlWwjxP8CTUsq/CSG2HuWefCBwEDkFONxGnY1SygYgVwixB10YvumkXZ1GSknFa/+HY+dOrCNGEHPzTdh+8hMlCG0QaY3kjMQzOCPxGCaFuZ2Q/w0YgyDldKgqgL+OAIMZUsbDYF98Ifl0MHQuzbWU0i8UjeLQoTB1UmACyzvTbnv3NR578TIgdIC/M298eo+0RBJubTpW4+mKvkZnh4a+Bp4EfgdcKKXMFUJkSylHdnCPEX3YZxpQgN65L5BS7gyoMxOYL6W8WggRA2wFRkspy9pr93hiBLKhgapVqyhd+hwNhw5hycggYdH9BJ9++jG1p+gEDQ596mruF3DgCz2egISLl8Lo+WAv0WcqJYzqs4n2FIqTga4YGroGuAF41CcCacD/dXSDz4O4BfgUffx/mZRypxDiYWCzlHKVr+w8IcQuwAPc1ZEIHC/CZCLikksInzOHqg8/pOzvS/2dj8deixYchFCdUddisur7NaRP18/ryiFvvb5XNMCulbB6IQRFQurZvhlJkyEmXU1VVSh6iE6nmBBCmIFhvtM9vuGcHqcrZw1Jr9ff8Rf+/gHqvt1CzA03EjZrJsKgUh/3CPZiOLBW9xZyv4CqQ4CAuw9AcJQ+jdUSChEDe9tSheKE5rg9At9MoVeAPPQg8AAhxNVSynVdZWRvEPj0H3LmmdRv/ZbDCxdS+uyzxNx4A2Hnn48wqlkX3YotTk+DMeoyPSdSRa6+0jk4Si//bBHs/7e+BWjaOXqMIfUcNVVVoehCOhsj2II+vr/Hdz4MeENK2eOD6925jkB6vdR8/i9KlyzBuWcP0b+6lrg77+yWz1J0kuLduseQ+4U+pOSshkFnwTWr9fK8/+r7NvTDZHoKxY+hK2IEpkYRAJBS7hVCnHQ7mwtNI2zGeYROPxf7f/6DJUNP1Fa/cyfO778nfM4chOmk+9p9m7gM/TXhBvC44ch3+iY9AM4aePUiPVVG0hjfqudzYMAEMAf3rt0KxQlEZyOjm4UQLwkhpvheLwBbutOw3kRoGqHnnos5Rd8AonrVKgrvX8T+GTOp+OdbSFf7KzkV3YjBqE87HTRRPzcGwVWr4Jy79OmpX/0NXpsL3/gWqjuq4OCGJuFQKBRt0tmhIQtwM3A2eoxgHbBESunsXvNa0xspJqSU1K5bR8mSJTi+244xMZHYW28lYu7FPWqH4ig47fDDRogdrudB2rkC3v45mEJ08WjMkZSQBZqaDKDoX6iNaboIKSW1//2K0mefJeTMM4n99S1IrxfpcqFZrb1ik6ID6iv1NQwHvoDcdVDqG9389bcQPQRKfOcxw9RUVcVJzzELgRDCt/qnbaSUPb6rSV9IOielBI8HYTRS/dlnHPnf/yX6l78kct48tKCgXrVN0QE1R/TtPkdcrHf8710H2/+pJ8xrjC+knQORg3rbUoWiyzkeIejwf4SUssezv/UFIQikfvt2iv+ymLqvv8YQHU30L35B5PzL0YJVsLLPU5Hnm5G0Tn/VlujewS2+DCcHv9L3YLCp7OiKEx81NNQD1G3eTOmSv1P71VdYR40i7a1/9rZJih+DlFCyG2pLIW0SeD3wRJoecI49pSlH0qCzIOjY9oxVKHqTrkhDXUPrIaIqYDNwp5TywHFb2Un6qhA0Urd1K157LbZJZ+N1OKh4fTkR8y7DYLP1tmmKH4PXC4VbdU/hwBd6ENpdD2fdDtMf0pPr5a2HgRPVVFXFCUFXCMFD6JlDl6PPGrocSAD2ADdKKad0mbVHoa8LQSA1//43+TffghYWRtTVVxH1s59hCAvrbbMUx4LbCfmbITRBDzTnfgmvXACaCQaM9616ngIp49SMJEWfpCuE4OuWG9ULITZKKScIIb6TUp7aRbYelRNJCADqs3dS+ve/Y//3v9FsNqKu+hkxN96oFqad6Ljq9BhCrm9GUuF3gISfr4bUs/TFbqZgJQqKPkNXrCz2+jaZf8d3fmlA2YkVZOhhgkZmMuDZZ3B8/70eQ9iwkZhf/xoAr8uFZlbbAp6QmIMh/Vz9BVBfoQ8hDZygn6/5A2S/ByMugsy5+mpnldlW0UfprEcwGHgK8C3pZANwB/o+A6dLKdd3m4UtONE8gpZ4nU40iwV3WRkHLppDxP/MJeqaazBGRfW2aYquZO+nsPU1yPkc3A4ITYQxV8JP7u9tyxT9lOP2CHzB4AvbKe4xETgZ0CwWQN8kJ2TCBMpefIny/3udyMsvJ/qXv8AYE9PLFiq6hGEz9JfTDns/0Vc51/m22pASvnhCn4mUMl55Copep7MeQQrwN/RN5iV653+blDK/e81rTVseQUNDA/n5+Tgcjp4257iRbjfemhq89fUgBMb4eLU5TiewWq2kpKRgOhFjLZWH4G+ng8cJoUkwYo4+fJQyTomCotvoimDx5+gzhl7zXboSuEJKOb3LrOwkbQlBbm4uoaGhREdHn7B7wXqdTrx2O8boaADcFRVoNhvaidjRdTNSSsrKyqipqSEtLa23zT3BwC8AACAASURBVDk2HNX68NHOFbDvc/C44LLX9JiC26nPRlKioOhCuiJYHCul/EfA+ctCiNuP37SuweFwkJqaesKKAOhDRoHDRg2HDwNgiIzEGBOjgsoBCCGIjo6mpKSkt005dqxhMOqn+stRDXs+hqHT9LKNS2DTiwGewliVC0nRrXT2kaNUCHGlEMLge10JdNvewsfCiSwCLREmE5b0dAwREXgqKnDm5OAqKEA29MruoH2Sk+nvjTUMTp0H5hD9PH6kniH1mxfgpXPhySz4/AE9tqBQdAOd9Qh+ATwD/BU9RvAV+ob2im5CM5sxJyfjjY3FXVqKt6oK4vScN1LKk6sjVDQnfbr+clTpnsLOFVC6t8kr2PQCJJ0GyacpT0HRJXTKI5BS/iClvEhKGSuljJNSXgz8TzfbdsJQWVnJkiVLjvn+J598krq6ujbLNLMZc1ISlmHDECYTUkpcuXm48vPxOnt8O4g2KS8vZ/r06aSnpzN9+nQqKirarDdz5kwiIiK44IILetjCExRrOJx6OSz4J1y+XL/mqIJPfwcv/gSeGqXv6VywRXkLiuPieKJRv+kyK05wulMIGhEG3wpVKdGCrHiqqvQhoz4gCI8//jjTpk0jJyeHadOm8fjjj7dZ76677uK1115rs0xxFBqf/K3hsHAPXPx3iM3Q4wkv/AS2vKyXe9xKFBQ/ms4ODbVFn/RJH/pgJ7sOV3dpmyOSwnjgwsx2y++9917279/P6NGjmT59On/605/405/+xFtvvYXT6WTu3Lk89NBD1NbWctlll5Gfn4/H42HRokUUFRVx+PBhpk6dSkxMDGvWrGnW9urVq/nNb35DTEwMp512GgcOHODDDz/k29xcbr/tNursdoIsFpYtW8aI007j5ZdfZuXKlXg8HrKzs7nzzjtxuVy89tprWCwWVq9eTVRUFFOmTGHMmDFs2bKFkpISXn31VR577DF27NjBvHnzeOSRRwC4+OKLOXToEA6Hg9tuu43rrruu1fd///33Wbt2LQBXX301U6ZM4Y9//GOretOmTfPXUxwHQZEweoH+qiuHPathqG+F8463YO3jepA582JIHK2GjxRH5XiEQD12+Hj88cfJzs5m27ZtAHz22Wfk5OSwadMmpJRcdNFFrFu3jpKSEpKSkvjoo48AqKqqIjw8nMWLF7NmzRpiWiwmczgcXH/99axbt460tDTmz5/vLztl5Ei+3LABA/DpihUseuQR3n3vPbwOB9k7drB12zYcDgdDhw7lj3/8I1u3buWOO+7g1Vdf5fbb9QlfZrOZdevW8dRTTzFnzhy2bNlCVFQUQ4YM4Y477iA6Opply5YRFRVFfX0948aN45JLLiE6Opprr72WG264gbFjx1JUVERiYiIAiYmJFBcX98CvrgAgOEpfsdxIaALEpMOGZ+C/T0Jkqi4KU+/X93xWKNqgw38Z7aSfBt0b6JNbcXX05N5TfPbZZ3z22WeMGTMGALvdTk5ODpMmTWLhwoXcc889XHDBBUyaNKnDdnbv3s3gwYP9c+Xnz5/P888/D+gicvXVV5OTk4MQgoaGBqSUeKqqmDRmDJaKCkJiYwkPD+fCC/VF4VlZWWzfvt3f/kUXXeS/npmZ6e/MBw8ezKFDh4iOjubpp59mxYoVABw6dIicnByio6N58cUXu/AXU3QZQ36iv+rKYfeHsHOlvvnOuQ/q5dvf1vd0TshSnoLCT4dCIKUM7SlDTiaklNx3331cf/31rcq2bNnC6tWrue+++zjvvPP4/e9/32E77bFo0SKmTp3KihUryMvLY8qUKQghMMbEEBQWhtdei7O6Gs3rxeRrR9M03G63vw2Lb92Cpmn+48B6a9eu5V//+hcbNmwgODiYKVOmtLl6Oz4+nsLCQhITEyksLCQuTu3o1esER8FpV+kvj+9v7nbBR78BZzVEDfYNH83Vp6sqUejXqKWLXUBoaCg1NTX+8xkzZrBs2TLsdjsABQUFFBcXc/jwYYKDg7nyyitZuHAh3377bZv3N5KRkcGBAwfIy8sD4J//bNr1rKqqiuTkZABefvll/3VhMKAFB2MZPgxjXBxSSqTLBXQsLG1RVVVFZGQkwcHB7N69m40bN7ZZ76KLLuKVV14B4JVXXmHOnDk/6nMU3UzjkJDRDLdugwufgohBsP5JWHq2PoQEepBZBZr7JUoIuoDo6GjOOussRo4cyV133cV5553HggULmDhxIllZWVx66aXU1NSwY8cOxo8fz+jRo3n00Ue5/349E+V1113HrFmzmDp1arN2g4KCWLJkCTNnzuTss88mPj6e8PBwAO6++27uu+8+zjrrLDweTyubhMGAKS4OYTKhheqOnbemBk91NZ6jzFBqZObMmbjdbkaNGsWiRYuYMGGCv+zaa6+lMdXHvffey+eff056ejqff/459957LwCbN2/m2muv9d8zadIkfvrTn/Lvf/+blJQUPv30087+xIquIiQaTv85XLUSFu6FC56E4bP1sn3/hmfGwX8egaKdShT6ESfFnsXff/89p5xySi9Z1L3Y7XZsNhtSSm6++WbS09O54447jqktd1kZ7uJipMeDZrNhjI3FEBLSxRb3HCfz371XyP0S1j2hb8EpvRCdrg8dnX2H2o7zJKCjXEPKI+jjvPDCC4wePZrMzEyqqqrajDt0FmN0NJZhwzDFJyAdDly5uTQUFXWhtYoTmrRJcPUHcOdemL1Yn4G07XUwWvXynM+h+PvetVHRLSiPoJ8ivV485eVowcFowcF4XS6ky4UWEnLCpK9Qf/ceoKEeTEH6MNFfM6G6QF/IljkXRlwMcRm9baGikyiPQNEKoWl6VtNg3eX3lJXhysvDlZuLp8b+owPLipMUk2+WuBDwqzVw/p8hOEZftLbkDPjXQ71rn6JL6FYhEELMFELsEULsE0Lc20G9S4UQUgjRplp1FS5HPR6P++gV+yHG+HhMiYnIhgZcB/NwHTiAp42ZTIp+TGg8jP8VXPMR3LlbF4Xh5+tlhdthyUR957WSvb1rp+JH021LDYUQBuBZYDqQD3wjhFglpdzVol4ocCvwdXfZAvpQSFXREQDC4+IxB6ngVyBC0zBGR2OIjMRTWYm7pARPdTUG34wjlfFU0YzQBF0UGnE7wBoBa/4Aax6FuEw9xcX46yAoovfsVHSK7vQIxgP7pJQHpJQu4E2grQnm/ws8AXTrPpNC04hISEIIQfnhAuwVZWr4ow2EpmGMisKSno4pIQEAb10drv0H8FRXq99M0TYDxsMvPobffA+zntD3WFj/JBh8O+z98DWU7utdGxXt0p1CkAwcCjjP913zI4QYAwyQUn7YUUNCiOuEEJuFEJuPZ1cqk8VCVMoArLZQ7OXlVBQexuv1HnN7jRxP9tHzzz+fysrK47ahqxGa5s94Kj1epNeD64cfcO3ff1RBkFJy6623MnToUEaNGuVfONeS3/3udwwYMACbzdYt30HRC4QlwhnXwy8+gTu/b9psZ/VCeOZ0+PvZsO7PULa/d+1UNKM7haCtcQR/7yGE0NA3urnzaA1JKZ+XUo6VUo6NjY09LqM0zUB4XDxhsXEYjMYuGe7oSAjaWuwVyOrVq4mI6NuusyHUhmXoUEzJyUivVxeEvLx2xeDjjz8mJyeHnJwcnn/+eW688cY261144YVs2rSpO01X9CbW8Kbj+W/CjMf09Qj/+V/422mw+i69zOuBDc/qG/D88DVU/qCnw1D0GN2ZjjAfGBBwngIcDjgPBUYCa32dcQKwSghxkZSy+fzQH8s/Zre+lnmxPqbpqkO8/lMCIwRSenEOuxjLWdch6srhraua33vNRx1+XMs01LNnz+ahhx4iMTGRbdu2sWvXrnbTOaemprJ582bsdjuzZs3i7LPP5quvviI5OZn333+foKDmuf3279/PFVdcgcfjYdasWSxevBi73Y7dbmfOnDlUVFTQ0NDAI488wpw5c8jLy/OvTN64cSOnnnoq11xzDQ888ADFxcW8/vrrjB8/ngcffJDc3FwKCwvZu3cvixcvZuPGjXz88cckJyfzwQcfYIqM5NGnn+aD99/HUV/PmZMmsXTpUmRtLZrN5hfV999/n6uuugohBBMmTKCystKfiyiQwJXKipOc8GSYeJP+qsqHXasgeqheVlsCn/629T3nPQJn/hpqS/W4Q2iS7nGEJenHEQPVQrcuojuF4BsgXQiRBhQAlwMLGgullFWAP++yEGItsPC4ReAY8Ho8OOtqqSssINxmxPAj72+Zhnrt2rVs2rSJ7Oxsf+bQ9tI5B5KTk8Mbb7zBCy+8wGWXXca7777LlVde2azObbfdxm233cb8+fNZunSp/7rVamXFihWEhYVRWlrKhAkT/NlF9+3bx9tvv83zzz/PuHHjWL58OevXr2fVqlX84Q9/YOXKlYAuMmvWrGHXrl1MnDiRd999lyeeeIK5c+fy0UcfcfHFF/PrX/+aBx54AICf/exnrHr7bWZkZvLSe++hhYZy4+23U1BQwIABTc8AKSkpFBQUtBICRT8lPEUXhEZs8XB3LlQfhprCpveUcXq5vVjPolpf3rydOUtgzBVwJBs+udcnEIlN74POhJDmqd0VbdNtQiCldAshbgE+BQzAMinlTiHEw8BmKeWq7vrsDp/gzcGtyg2Auaaa6tISylwuwue9hSX4+FIvjB8/3i8CQLvpnANJS0tj9OjRAJx++un+ZHOBbNiwwd9xL1iwgIULFwL6uPxvf/tb1q1bh6ZpFBQUUORbNZyWlkZWVhYAmZmZTJs2DSEEWVlZzT5j1qxZmEwmsrKy8Hg8zJw5E6BZvTVr1vDEE09QV1dHeXk5I0aM4IIZM7h2/nyk04lr3z68vpTYgagZR4p2EULPlhocBQkjW5fHj4B7cqHBoQtEo1g0CoXbCR4XHNygl3kb9OtXvQ+Dp8D3H+jDUI0i0SgUo6/Qp8Q69eSQWPpvrKpbd6qQUq4GVre41mbeZSnllO605WgEhYZhslipLDpCReFhIhISsYYc+z+MkIAcPp1N5xyYCtpgMFBfX9/pz3v99dcpKSlhy5YtmEwmUlNT/Z/RMsV0YPrp9tJSm0wmf+fdWM/hcHDTTTexefNmBgwYwIMPPojT6cQYEYEhPBxPVRXukhKSIiI59MMP/nbz8/NJSkrq9HdRKNrEZIWoNP0VSMrp8MvP9GOvF+rKoOawnmobdI9jyDT9Wtl+yPtS3/v5lIt0Ifj2Vfj0PrCE+cQiUR96mvGoLk7lueCo1K+FxIJ28q3DVVsWBWA0m4lKTqGuqgKLb51BZ+bPt5dGupHOpnPuDBMmTODdd99l3rx5vPnmm80+Iy4uDpPJxJo1azh48OAxf0Z7NApLTEwMdrudd955h0svvRTQn/gbBWHO/MtZ8txzXH755ax//33CbDYSfFNRFYpuRdPAFqu/GhkwXn8F4qptyqE0aKK+cU91oS4W1YVQsrap/JsX9R3fADSTvoYiNBGuWa1Pj81dBzVFPgHxeR2mPrlvV7soIWiBpmnYIvUhG6/HQ8WRw4RERHboHQSmoZ41axazZzcPVs+cOZOlS5cyatQohg8fflxB0ieffJIrr7ySv/zlL8yePduflvqKK67gwgsvZOzYsYwePZqMjK7PARMREcGvfvUrsrKySE1NZdy4cf6yxnjFDTfcwAVz5vCJLy11kMnE0ocfxpmTgzEujrFTpvhjKXfffTfLly+nrq6OlJQUrr32Wh588MEut1uhaIU5YOg3aYz+ao+xv4CBE5vHLxzVTWskNi/TZzwFEjUYbt2qH3/zEtiLmoLcYYkQlqx7G30ElXSuAzwNDVQWFdLgdBISHoEtOhp91mvvUVdXR1BQEEII3nzzTd544w3ef//9XrWpI6SUeKurcZeU4HU4EGYzlsGDEcbjfwZRSecUfQJnjS4QgcFuKWGyb3rs8nmw91Oa7fobnwU3rtePP7hdH6ryB7sT9RTgiaO61MyOks4pj6ADDCYTUUkp1JSXUltVicvpIDwuAaPJ1Gs2bdmyhVtuuQUpJRERESxbtqzXbOkMQggM4eFoYWH6xjg1NeBbqOatq0P4RE2hOGGxhOr7QMcOb7t8wT/B06B7BY3DT1pA1+uogsJtsOdjcPvigsNm6veBvoscQs/rNPW+bvkKSgiOgtA0wmLiMFuDqS4poqa0hMjE3gt8Tpo0ie+++67XPv9YEUJgCAvDEBYGgHS7cebmIUxGfYOc8HDESRiEUygAfRgpPEV/teSn/9DfpdSD0tWHIXDkIWU8VB1qfq2LUULQSaw2G0aLxb/Ht9fjQQihOq9jxWDAPCAFd3ExDQUFuItLMMbGYIiIUL+pon8iBARF6q9ALljc7R+thOBH0DgkJKWkqvgIXo+H8PgEjCZzL1t24tHoIWihoXhranCXlNBw+DBacDDCau1t8xSKfoV69DoGhBAEhYXjcTdQln+IervK23+sNAqCefBgLEOGoPlEoKGwEHd5ObILkgIqFIqOUR7BMWINsWEyW6gsPkJV0REa6uuxRcegqWGNY0IIgfDlVZJeL966erxlZbhLSvQYghoyUii6DfU/6zhonFXUIDT+/txSpDy2p9cnn3ySurq6Lrau53j77bfJzMxE0zRaTu0N5JNPPmH48OEMHTqUxx9/vN16QtMwD07DPGgQwmii4fBhnDk5eGpP3N9IoejLKCE4ToQQeDQDr735FgaDESkljtraH9XGiS4EI0eO5L333uOcc85pt47H4+Hmm2/m448/ZteuXbzxxhvs2rWr3fpCCAyhobogpKYiLBY0sx6j8bpcashIoehCTrqhoT9u+iO7y3d3aZsZURncM/6edssD01BPOWcS9952Ky+8+hqrVn+M0+lk7ty5PPTQQ9TW1nLZZZeRn5+Px+Nh0aJFFBUVcfjwYaZOnUpMTAxr1qxp1vbq1av5zW9+Q0xMDKeddhoHDhzgww8/ZNOmTdx+++3U19cTFBTEP/7xD4YPH87LL7/MypUr8Xg8ZGdnc+edd+JyuXjttdewWCysXr2aqKgopkyZwpgxY9iyZQslJSW8+uqrPPbYY+zYsYN58+bxyCOPALSbPjuQzizq2rRpE0OHDmXwYD3/y+WXX87777/PiBEjOrxPCIHBZsPg27xGSknDoXxkgwuv3Y7X6UQLyKWkUCh+PMoj6AIef/xxhgwZwrZt2/jrU0/z9bbv2LNnD6vffZtvNn3Nli1bWLduHZ988glJSUl89913ZGdnM3PmTG699VaSkpJYs2ZNKxFwOBxcf/31fPzxx6xfv57A3dkyMjJYt24dW7du5eGHH+a3v23K556dnc3y5cvZtGkTv/vd7wgODmbr1q1MnDiRV1991V/PbDazbt06brjhBubMmcOzzz5LdnY2L7/8MmVlZYCePnvLli1s3ryZp59+2n/92muv7XAYqCXtpaY+FozxcQizGU91NfvPnU75q6/ibSOJn0Kh6BwnnUfQ0ZN7TyCE4MsNG/lyw0amzjwfkNQ7neTk5DBp0iQWLlzIPffcwwUXXMCkSZM6bGv37t0MHjzYn856/vz5PP/884CeZO7qq68mJycHIQQNDQ3++6ZOnUpoaCihoaGEh4dz4YUXAno66e3bt/vrNe5XkJWVRWZmpn+/gMGDB3Po0CGio6PbTZ/94osv/qjfpa1UJseyojjQQzBUV2NOS6PoD4+B0UjUggVHb0ChULTipBOCvkDj3gDX/vKXVJcUY4uKxuQbvtiyZQurV6/mvvvu47zzzuP3v28zK7e/nfZYtGgRU6dOZcWKFeTl5TFlyhR/2bGknW55j9vt7nT67M6QkpLCoUNNW1h3RWpqzWJh0KuvUPfNN1h9+y1Ur15Nw5EiIi+fhxasdq9SKDqDGhrqAlqmoZ4xYwbLli2j3uEgMjGJ4tJSiouLydmZjclg4Morr2ThwoX+Td3bS2OdkZHBgQMH/JvC/POf//SXVVVVkZycDMDLL7/cLd+rK9Nnjxs3jpycHHJzc3G5XLz55pt+j+R4CR43zr/+wP7f/1L8xBPsm34eZS+9hPcEDsIrFD2FEoIuIDAN9V133cV5553HggULmDhxIllZWVx66aVUVlbw7ZYtnHHGGYwalcWjjz7K/fffD8B1113HrFmzmDp1arN2g4KCWLJkiX/P4fj4eH/a6bvvvpv77ruPs846C4/H0y3fa+bMmbjdbkaNGsWiRYuapc8OjBGsWLGClJQUNmzYwOzZs5kxYwYAhw8f5vzzzwfAaDTyzDPPMGPGDE455RQuu+wyMjMzu9zmpEcfZdDy17EOH07xn/7MvmnnUtWHs7MqFH0BlYa6B/G43VSXFOGsq8NqsxEWE4dm6HiHZLvdjs1mQ0rJzTffTHp6OnfccUcPWdy3OdrfvW7rVkqfXULET39K2Izz8NbWIiUYbMe3DalCcSLSURpq5RH0IAajkYiEJGxR0Thq7VQUFnQYBwB44YUXGD16NJmZmVRVVXH99df3kLUnPsFjxjDwxRcIPW86AGX/eJn906ZRuvQ5PHZ7L1unUPQdVLC4hxFCYIuMwmwNwuvVM5g2ikFbs2juuOMO5QEcJ42/q23yZBw7dlDy5JOU/eMfRP/8aiKvvBJDaGgvW6hQ9C7KI+glzEFB/u0v66urqCrSs5kquo+grJEMeG4pqW+/RfCYMZQ89TSFi9qftaVQ9BeUR9BHcNbVUlZwiPC4BMwqDXO3EpSVxYClf6c+eyeaRU8h7srPp2rFSqKuvsq/eY5C0V9QHkEfIDg8gsgkfeeiisP51FZWHDV2oDh+gkZmYklPB8C+bh2lzz7LvmnnUvL03/BUVfWydQpFz6GEoI9gtlqJTh6AJTiEmrJSGpzO3japXxG1YAFpK94jZMIESpcs0QVhyZLeNkuh6BGUEHQBlZWVLDnGTuP888+nsrISAM1gIDw+gajkFP/wkCdgJfCJwu7du5k4cSIWi4U///nP7dbLzc3ljDPOID09nXnz5uFyuXrQytZYTzmFlL89Tdr7Kwk56yzcRcX+MjXLSNGbSLe7W/NpKSHoAjoSgqMt9lq9ejURERH+cyEEZqu+QYurvp7SH/KwV5SfUENFUVFRPP300yxcuLDDevfccw933HEHOTk5REZG8tJLL/WQhR1jHT6clKeeJOEBPZBct3Ur+86ZTPFfFuOuqOhl6xQnElJKvPX1NBQX4zyQS/327Xh9aeqdOTmUv/IKJc88S9Fjj3P4/vvJv+123L7kkhX/fIuccyaz+7TT2T0yi8L7F3WbnSdlsPjgz65qdS101kyiFizAW1/Poetaz8UPnzuXiP+Zi7uigoJbb2tWNui1V1vVDyQwDfX06dOZPXs2Dz30EImJiWzbto1du3a1m845NTWVzZs3Y7fbmTVrFmeffTZfffUVycnJrHjvPSwhIdjLy3A56gmPjSfv4EGuuOIKPB4Ps2bNYvHixdjtdux2O3PmzKGiooKGhgYeeeQR5syZQ15enn9l8saNGzn11FO55ppreOCBByguLub1119n/PjxPPjgg+Tm5lJYWMjevXtZvHgxGzdu5OOPPyY5OZkPPvgAk8nEww8/zAcffEB9fT1nnnkmzz33XKtpr3FxccTFxfHRRx+1+5tJKfnPf/7D8uXLAbj66qt58MEHufHGGzv8rXuSxh3RjJGR2KZMpuzFFyl//XWirlhA1DXXYIyK6mULFd2J9HrxVlfjsdfitdfgtdvx1NRgHTECU3w8rrw8Kt99D2+tHU+NXU+LXlND3D13E5SVRfVnn1HwmzuhhVef+uYbBI0eTf32HRQ9pm/QpAUHo9ls+h7etbUQG4spKYmQcyZhsIWihdqwZmR023c9KYWgp3n88cfJzs5m27ZtAKxdu5ZNmzaRnZ3tzxy6bNkyoqKiqK+vZ9y4cVxyySVER0c3aycnJ4c33niDF154gcsuu4wVK1dyxRVXYA6qpqa0hLKCQ/z6llu47bbbmD9/PkuXLvXfa7VaWbFiBWFhYZSWljJhwgR/Lp99+/bx9ttv8/zzzzNu3DiWL1/O+vXrWbVqFX/4wx9YuXIlAPv372fNmjXs2rWLiRMn8u677/LEE08wd+5cPvroIy6++GJuueUWf6K8n/3sZ3z44YdceOGFfltuuOGGTv1mZWVlREREYDTq/wSPJy11d2NOTSV58WJibrqJ0r8vpezFl6j64EOG/utzhFH9F+qrSLcb1w+H8NbqHXRjZ20dcQrWU07BXVJC8VNP4fVd99hr8NbYibnpJsIvmI0jO5u8y+a1ajfpz38m/ILZNBQVU7ZsGQZfB66FhmIICQHfKIAlLY3oX/wCzWbDEGpD83XoZl+fEHb+LELPnYZmsyHayDBgm3Q2tklnd++P5OOk/Ffc0RO8FhTUYbkxMvKoHkBnGD9+vF8EgHbTOQeSlpbG6NGjATj99NPJy8tDCEFwWDgmi5Wq4iN8/fXXrPrgAwAWLFjgH35pzHi6bt06NE2joKCAoqIif7tZvuycmZmZTJs2DSEEWVlZ/oR2ALNmzcJkMpGVlYXH42HmzJkAzeqtWbOGJ554grq6OsrLy8nMzOTCCy/stAA00lVpqXsSy9ChJP/lz8TcfBOu3FyE0Yj0eil76SUi5s7FGBPT2yae8EiPB9nQ4E8iWLdlC57qary+p3KP3Y512DBskycjXS7yb7vd14nrnb3XbidywQJib/01nupqDvhyXQUSe9utWE85BenxULvuS9+TuA2DLRRTQiIG31CtKSWF+Pvu1cttob7O3IZ50CAAgsePIyN7R7v/bi3p6cT9pv3FoFpQEPj26e5tTkoh6AuEhDTls+lsOufAVNAGg4H6+nr/ucliISp5APj+0bnq65sFkl9//XVKSkrYsmULJpOJ1NRU/2ccS1pqk8nk/wfeWM/hcHDTTTexefNmBgwYwIMPPnjMaaljYmKorKzE7XZjNBq7JC11T2EZPBiLb6c1R3Y2JX99GsosBAAAHbxJREFUktJnlxA+92JMcXEARMybhzEqivrvvsO+7kv9RikBXQCjfv5zDGFh1H69idr//repzCeQMTffjBYURM3atdRt/Npf3iig8ffcgzAYqP74Y+q+acy9pZcLg5GE+38HQOW771G/bVvTvVKiBYeQ8Dt9I6PyV1/DsXNnQDkYoyKJv+8+AEqXPodz7x6f+Xq5KSmJ+LvvAqD4L4tx5eX6y5ASy+A04nwPKIUPPUTD4cO6/b5ya2YmcXfcDsAP119Pw6F8f2cu6+oImz2b5L/okwwO/eq6Vhlkwy+9BNvkyWAy4S4qQgsJwZSQgJY+FIPNhnWknszQEBZG0p/+5OvkfU/tITaMUZEAmBISSF/3Rbt/Z2NUFFFXX91ueV9/cPkxKCHoAtpLI91IV6Vz1jSNCRMm8M47bzNt4gReeX15s8+Ii4vDZDKxZs0aDh48eEyf0RGNnX5MTAx2u5133nmHSy+99JjaEkIwdepU3nnnHS6//HJeeeUV5syZ05Xm9ghBo0Yx+KMPKVv6HJVvve0fFgg991y/EJQ++2zzm4Qg4pJLMISFUb9tG2XLloEQCF8ZQhB97bUQFIRj+3Yq3nqrWRlCEL9wIRgM1GdnU/3RR/4HBIRAmM1+IXDs2kXN2jUImu41BExOcO7bR9033zRr25SQ4C93HTyIY9f3zdqXAZsgNRQW4so72Oz+wAV5nopKPKVlzcoDO3ZjVDRaUDCaLaTNsfABzy1FWK1oIb7hldBQhO+BRQhB2nvvtvu3EUYj4RdecLQ/oQIlBF1CYBrqWbNmMXv27GblM2fOZOnSpYwaNYrhw4c3S+f8Y3nyySe58sor+ctfvPxk0tmE2kKoKS9jwYIFXHTRRf/f3pmHR1Wlefj9aslGQkISwpIEAgRDJwZBE2SRHmkd2ZqgMw4gwtgtoDTyqNja3Tii4GgjojR2q4Ng49KNRAGRRWxwwUk7LUYQhIi2YTWELQRICEmoVNWZP+5NUVkqBEllPe/z5Mmt+517zndP1b3fPefc8zukpaXRr18/+vhhYCkiIoJp06aRmppKQkIC6enpHpv3GMHx48dJS0ujuLgYi8XC4sWL2bt3L+3bt2fUqFG8+uqrdO3alQULFjBhwgQee+wx+vfvz5QpUxrc58YgsEcPui54hi5PPwVut3HDM8cOOkyeTIfJk30+PUbfew/R99ZcB7qSjvffT8f77/dp7/TII3R65BGf9s5zHqPznMd82rs8Oc+nDaDr/N/XaY99bmGd9rjFf7ii/EO8fmMa/+FXGWoRGQG8AFiBV5VSz1SzPwRMBZxAAXC3UqrOR9mWLEPdEJSWlhIcHIyI8NZbb/HXN99k+Ut/IiAomIguXbFY2s4bwW3pe9dorpS6ZKj91iIQESvwEvCvwBHgSxFZr5Ta65VsJ5CmlCoVkV8BzwI1h+k1Hnbs2MHMmTNRShEREcHy5csJj4nBUV7WqvosNRpN4+HPrqEBwD6l1AEAEckExgKeQKCU2uqVfhswyY/+tAqGDh3K119/XWN/cJjRL+t0OCgrOUdoh0gdGDQaTb3wZz9CLJDn9fmIuc8XU4APajOIyD0isl1EtheYs+40tXOhtITzZ05z+ugRXF6DehqNRuMLfwaC2h5Hax2QEJFJQBpQ68iTUmqpUipNKZXWsWPHBnSx9dEuIpLwTp1xOhwU5v9A+XmtkaPRaOrGn4HgCBDv9TkOOFo9kYjcDPwXkKGU0pKbDUBwaBhRcfFYbXbOHj9GuRZM02g0deDPQPAl0FtEeohIADABWO+dQET6A69gBIGTteSh+ZHY7AFEdo0jLCqKwJAQoPbZvBqNRuO3QKCUcgIzgc3At8A7SqlvRORJEckwky0EQoFVIrJLRNb7yK5ZcyUy1GDMDSitNnuyIRCLhXYRkYjFgtvl4nR+nl9aB6tWrSIlJQWLxUL1V3vnz59PYmIiSUlJbN68udbjm5sctUbT1vDrS+dKqU1KqauUUr2UUk+b+x5XSq03t29WSnVSSvUz/zLqzrF50lwDgTdKGROdzp44RvGpkyi3u8Hyvvrqq3n33Xf56U9/WmX/3r17yczM5JtvvuFvf/sbM2bMqFWWu7nKUWs0bYVWN7P47+98z6m8hn3qjY4PZei4q3zaq8tQL1y4kIULF/LOO+9w4cIFbrvtNubNm8f58+cZN24cR44cweVyMWfOHE6cOMHRo0cZNmwY0dHRbN26tUremzZt4qGHHiI6Opprr72WAwcOsHHjRrKzs3nwwQcpKysjODiY1157jaSkJF5//XXee+89XC4XOTk5/PrXv8bhcPCXv/yFwMBA3v7rm1BUxMgxY0lLT2fnrl0UFBTw5ptvMn/+fPbs2cP48eN56qmnAHzKZ3vja1LXunXrmDBhAoGBgfTo0YPExESys7MZNGiQJ01LkKPWaFo7rS4QNAXVZai3bNlCbm4u2dnZKKXIyMggKyuLgoICunbt6tHpLyoqIjw8nEWLFrF161aiq6lXlpeXc++995KVlUWPHj244447PLY+ffqQlZWFzWbjo48+4tFHH2XNGkN3JScnh507d1JeXk5iYiILFixg586dzJo1i7UbNzF92lRDBMzlIisrixdeeIGxY8eyY8cOIiMj6dWrF7NmzSIqKsqnfPbUqVOZPn06aWm1TlQEID8/v4qcRm1S0y1Jjlqjaa20ukBQ15N7Y7Flyxa2bNlC//79ASgpKSE3N5ehQ4fy8MMP89vf/paf//znDB06tM58vvvuO3r27OmRs77jjjtYunQpYASRu+66i9zcXESECq85A8OGDSMsLIywsDDCw8MZM2YMYMhJ7969m6B2odgCA/n3ceMASElOJjk5mS5dugDQs2dP8vLyiIqK8imf/eqrr16yHuojNd0S5ag1mtZGqwsEzQGlFLNnz+bee2uuhLZjxw42bdrE7NmzueWWWzyLvPjKxxdz5sxh2LBhrF27lkOHDnHjjTd6bPWRnRYRQtq1QylFWfFZLErhdDiwBQR40tVXPtsXcXFx5OVdnFNYm9R0S5aj1mhaC21HocyPVJehHj58OMuXL6fEfEMnPz+fkydPcvToUUJCQpg0aRIPP/wwX331Va3HV9KnTx8OHDjgWRTm7bff9tiKioqIjTUmar/++us/2ncRIbBdKApFYf4PlJ0rrlLGlchnZ2RkkJmZyYULFzh48CC5ubkMGDCgRvmVctRAi5Wj1mhaMjoQNADeMtSPPPIIt9xyCxMnTmTQoEGkpqZy++23c+7cOfbs2cOAAQPo168fTz/9NI89ZsgD33PPPYwcOZJhw4ZVyTc4OJiXX37Zs+Zwp06dCA8PB+A3v/kNs2fPZsiQIbW+iXM5BAQFExAUbK6CdgJnhQO3282IESNwOp307duXOXPmVOnvnzp1qudV0bVr1xIXF8fnn3/O6NGjGT58OGCshjZu3DiSk5MZMWIEL730ElZzSb5Ro0Zx9Kgxv3DBggUsWrSIxMRECgsLW6wctUbTUvGrDLU/aGsy1CUlJYSGhqKU4r777qN3797MmuV7+bsrQSnF+TOnKTtXTGRcPFZr8+45bM3fu0bT0NQlQ61bBM2cZcuW0a9fP1JSUigqKqp13KGhEBFCI6OIiu+O1WrD7XZz6ofDnD1+jJLThZSXnMPpcOgZyhpNK6N5P/JpmDVrlt9aAL6oXNxGud3YAgJwVjiqiNeFRXekXXgELmcFZefOYQ8INAaZbTb9xo9G0wLRgUDjE6vNRkRn45VSt9uNq8KB0+HAHhQEgPOCg5LThZ70YrFgCwigfXRH7IFBuF0uFKrZdzFpNG0dfYVq6oXFYsESGIQ9MMizL7BdO2ISeuJ0OHBWOHA6LuB0OBCzRVFeco7iUwVYrFZsZqvBFhBAUGhYm1pSU6Np7uhAoLkiLFYrAcHBBAQH17DZg4MJi4o2AoXDQVlxEUopgkLDADh/9gyO8jIzQARisxuBQncvaTSNiw4EGr9hDwjEHnBxcptSCpfT6dUaULgqKrhQet6zZJHFaqVj9x6IiGdcwhYQiFWPP2g0fkO3zxuAK1EfHTVqFGfPnm1gj5qW7777jkGDBhEYGMhzzz3n2S8ifPTxxyQlJZGYmMifliwlOr47nRJ6ERXXjfCYzoRGRuFwOBg/fjzJKVczeMgQdv7jM04eOkBhfl6VMYmGVFDVaNoyukXQAFQGghkzZtSwuVwuzySq2ti0aZM/XWsSIiMj+eMf/8h7771XZb/L5eK+++7jww8/JC4ujvT0dDIyMkhOTsYeGIjdlMJ4+eWX6dChA/v27+etFStY8MKfWL70FZyOC7i9Js+VnDnNy9PuJDq+O9HduhMd353YpGSi4ro16vlqNC2dVhkI3p73uxr7kgYOpd/w0VRcKOfdZ+bWsKf8y81cfePNlBYXseEP86vYxj/xTJ3lVZehHj16NPPmzaNLly7s2rWLvXv3+pRzTkhIYPv27ZSUlDBy5EhuuOEG/vGPfxAbG8u6desIrtb3vn//fu68805cLhcjR45k0aJFlJSUUFJSwtixYzlz5gwVFRU89dRTjB07lkOHDnlmJm/bto1rrrmGX/7ylzzxxBOcPHmSFStWMGDAAObOncvBgwc5duwY33//PYsWLWLbtm188MEHxMbGsmHDBux2O08++SQbNmygrKyMwYMH88orr9TosomJiSEmJsajslpJdnY2iYmJ9OzZE4AJEyawbt06kpOTq6Rbt24dc+fOxWK1MmHiRB586CHCoqKrlKOUIjAkhF7XXU9h3mFyPvmQigvlXDf6Vm78z6k4HQ42LH6G6LhuRHdLIDq+O5GxcVht9jq/S42mLdIqA0FjU12G+tNPPyU7O5ucnByPcqgvOWdvcnNzWblyJcuWLWPcuHGsWbOGSZMmVUnzwAMP8MADD3DHHXewZMkSz/6goCDWrl1L+/btOXXqFAMHDiQjw1jnZ9++faxatYqlS5eSnp7OW2+9xWeffcb69ev5/e9/73ly379/P1u3bmXv3r0MGjSINWvW8Oyzz3Lbbbfx/vvvc+uttzJz5kyPUN7kyZPZuHEjY8aM8fgyffp0n/WUn59PfPzFZazj4uL44osv6kxns9kIDw+nsLCwiky3iBAQHMLw6fcDRjdR8amTiMVofZUWn6X45AkO7drhaUVYrFZunnYfqcNuoazkHEe+zSE6vjsRMZ09bzppNG2RVhkI6nqCtwcG1WkPaR9+yRZAfRgwYIAnCAA+5Zy96dGjB/369QPguuuu84jNefP55597btwTJ07k4YcfBown5EcffZSsrCwsFgv5+fmcOHHCk29qaipg6P/cdNNNiAipqalVyhg5ciR2u53U1FRcLhcjRowAqJJu69atPPvss5SWlnL69GlSUlIYM2ZMnQGgkvpKTv8YaWqxWAiP6ez53D46hrueewmXs4IzR/M5lXeYU3k/ENPdaI0c+/471j/3NAC2wECiYrsRHd+dAbfeTmTXONxuFyIWPUCtaRO0ykDQHGjXrp1nu75yzt7y0VarlbKysnqXt2LFCgoKCtixYwd2u52EhARPGfWRpfZOZ7FYsNvtnptgZbry8nJmzJjB9u3biY+PZ+7cuQ0uS+2dLi4uDqfTSVFREZGRkfUuxxurzW50DXVLqLI/PiWViU8/z6kfDptB4jAHd20nbcxtAOR88iF/X/lGlfGHqPjudEm8qlV2Lym3G6UUFqsV5XZTcvY0rgonLmcFrooK3E4nIREdaB/dEafDwaHdO839FbicRrrOiUl06tGL0uIidv5to2kz7RUV/OSGG4lP6cuZ40f59I1luJxORASxWBCLhbTRtxKf0pfC/Dy2rclELBYsps1isXLNLaOISehJ4ZEf2PPJFi+7FYvVQsq/3ER4TGcK8/M48NWXnmMr0/UeMJiQ8AjOHD/K8dx/mjarJ123q/sSEBxCccFJzhw/6nW8kSamR0+sNjulRWcpO1dcJW+xWAiLjEYsFirKy3E6K7B45S0WCxartdk+WOhA0AD4kpGu5ErlnL0ZOHAga9asYfz48WRmZlYpIyYmBrvdztatWzl8+PCPLsMXlTf96OhoSkpKWL16Nbfffnu9j09PTyc3N5eDBw8SGxtLZmamZ4lKbzIyMnjjjTcYNGgQq1ev5mc/+1mDX0D2wCC6JCbRJTGpyv7K1kiHrrH0vn4wp/IOszfrExxmUL7vz5lYQ+18+/etnDx8kI7dEoiK705UbDy2gACf5bndLtwuNza7EUTOnz1DRXm55ybqclZgCwikoxmwDu/eRfn5c56bqMvpJLRDJInphgLslxvepay4qIo9pkdP+g//OQAbFs2n/HwJLqcTt9OJ01lBr+uuZ8i4OwFYcu9knBUOz/HK7ebaUWMZdtc0nBUOlv7qFzXOYeC/jWfI+MlcKD3PuoX/XcM+dOIv6NSjF47SUratWYnVbsdqs2GxGf9j+6QYdex2c/7sGSwWKwplBCG3osJxAQBHWSnH93+Pcrtxu93Gf5eL3gOMJU6LTxWw++PNKJfLY1fKTdxPUgmP6czJQwfI+uvyGv516tmbkPAI8nJ28+GyF2vYf/mHJUQGh/D9ts/431qOv+d/XicsMppdWzbx+eqav9uZr71DYEgI/7dqBTs2rq1hfyhzAwAf/fl/yPlkM2KxmgHCQkBwCPe89BoAn7z+Cvu3Z1cJIu3CIxj3xPwaeTYUOhA0AN4y1CNHjmT06NFV7CNGjGDJkiX07duXpKSkKnLOl8vixYuZNGkSzz//PKNHj/bIUt95552MGTOGtLQ0+vXrR58+fa7onGojIiKCadOmkZqaSkJCAunp6R6b9xjB8ePHSUtLo7i4GIvFwuLFi9m7dy/t27fnxRdfZPjw4bhcLu6++25SUoybw+OPP05aWhoZGRlMmTKFyZMnk5iYSGRkZJWA528qA058cirxyUZ3mlKKc6cKOJ2fR1BoKADHD+xj1+b3cbsqF/qx0DXpJ0yYtwCADX94hsO7d1680So30d0SuGuhcQNa99xTHMv9Z5Wyu/ROYuJTzwPw6ZvLOJVXNZh379vfEwh2bX6f82dPYzVvsla7HYv14jhH+fkSnA4HVrsde2AgwTYbwWHtPfbe1w9GLBbP8Rarja69jaBotdv513tmXszbZsdqtxPR2Wi9BYe1Z9L8xZ5yrTY7FpuNwJAQAMI7deahzA0+g3dk1zgmzV/s8zvokpjElBeW+bT36Hcd97+xqso+7+7EpIE30Ova9ItBwgwowWHGRMakwUOJS041bReDSfvoGMM+5Kd07nWV1/Eu3MpNcKhRf1cNHEJkbFyVvCt1uQAS0wfSPioat3egcrs99dE99RoCgoMv5u1yY/F6szAqNp4LJSVVjg0IudjD4A+0DHULo7S0lODgYESEzMxMVq5cybp165rarSahqb93l9PJ2ePHOJV3iFN5hykvOcdNd/8KgJ2bN3LmWL55M7VjtdtoF9GBvjcZ4y77d2RTXnLO89RstdkJCg2j61VGAD999AjK7cZis3nstoBAz81WKdVsuxk0zZO6ZKh1i6CFsWPHDmbOnIlSioiICJYvr9mE1TQOVpuNqLh4ouLiSRpUdf3pyi4aX/S6bkCd9siucXXadRDQNCQ6ELQwhg4dytdff93Ubmg0mlZEq3l5uqV1cWmuDP19azQNR6sIBEFBQRQWFuqbQxtBKUVhYSFBQUGXTqzRaC5Jq+gaiouL48iRIxQUFDS1K5pGIigoiLi4uvvRNRpN/WgVgcBut1eZxavRaDSa+tMquoY0Go1G8+PRgUCj0WjaODoQaDQaTRunxc0sFpEC4McK6UQDpxrQnYZC+3V5aL8un+bqm/br8rgSv7orpTrWZmhxgeBKEJHtvqZYNyXar8tD+3X5NFfftF+Xh7/80l1DGo1G08bRgUCj0WjaOG0tECxtagd8oP26PLRfl09z9U37dXn4xa82NUag0Wg0mpq0tRaBRqPRaKqhA4FGo9G0cVp1IBCR/xCRb0TELSI+X7kSkREi8k8R2Sciv2sEvyJF5EMRyTX/d/CRziUiu8y/9X70p87zF5FAEXnbtH8hIgn+8uUy/fqFiBR41dHURvJruYicFJEcH3YRkT+afu8WkWubiV83ikiRV3093gg+xYvIVhH51rwWH6glTaPXVz39avT6MssNEpFsEfna9G1eLWka9ppUSrXaP+AnQBLwKZDmI40V2A/0BAKAr4FkP/v1LPA7c/t3wAIf6UoaoY4uef7ADGCJuT0BeLuZ+PUL4MUm+F39FLgWyPFhHwV8AAgwEPiimfh1I7CxkeuqC3CtuR0GfF/L99jo9VVPvxq9vsxyBQg1t+3AF8DAamka9Jps1S0CpdS3Sql/XiLZAGCfUuqAUsoBZAJj/ezaWOANc/sN4FY/l1cX9Tl/b39XAzeJ/9dKbIrvpV4opbKA03UkGQu8qQy2AREi0qUZ+NXoKKWOKaW+MrfPAd8CsdWSNXp91dOvJsGshxLzo938q/5WT4Nek606ENSTWCDP6/MR/P+D6KSUOgbGDxKI8ZEuSES2i8g2EfFXsKjP+XvSKKWcQBEQ5Sd/LscvgH83uxNWi0i8n32qL03xm6ovg8wuhw9EJKUxCza7L/pjPOF606T1VYdf0ET1JSJWEdkFnAQ+VEr5rLOGuCZb/HoEIvIR0LkW038ppdbVJ4ta9l3xO7V1+XUZ2XRTSh0VkZ7AJyKyRym1/0p9q0Z9zt8vdXQJ6lPmBmClUuqCiEzHeEL6mZ/9qg9NUV/14SsMvZkSERkFvAf0boyCRSQUWAM8qJQqrm6u5ZBGqa9L+NVk9aWUcgH9RCQCWCsiVyulvMd+GrTOWnwgUErdfIVZHAG8nyTjgKNXmGedfonICRHpopQ6ZjaBT/rI46j5/4CIfIrx1NLQgaA+51+Z5oiI2IBw/N8FcUm/lFKFXh+XAQv87FN98ctv6krxvtEppTaJyMsiEq2U8qu4mojYMW62K5RS79aSpEnq61J+NVV9VfPhrHntjwC8A0GDXpO6awi+BHqLSA8RCcAYePHbGzom64G7zO27gBotFxHpICKB5nY0MATY6wdf6nP+3v7eDnyizFEqP3JJv6r1I2dg9PM2B9YD/2m+DTMQKKrsCmxKRKRzZT+yiAzAuP4L6z7qissU4M/At0qpRT6SNXp91cevpqgvs6yOZksAEQkGbga+q5asYa/Jxh4Rb8w/4DaMyHkBOAFsNvd3BTZ5pRuF8dbAfowuJX/7FQV8DOSa/yPN/WnAq+b2YGAPxtsye4ApfvSnxvkDTwIZ5nYQsArYB2QDPRvp+7uUX/OBb8w62gr0aSS/VgLHgArz9zUFmA5MN+0CvGT6vQcfb6w1gV8zveprGzC4EXy6AaPLYjewy/wb1dT1VU+/Gr2+zHL7AjtN33KAx839frsmtcSERqPRtHF015BGo9G0cXQg0Gg0mjaODgQajUbTxtGBQKPRaNo4OhBoNBpNG0cHAo2mFkSk5NKp6jx+tTkjHBEJFZFXRGS/qSaZJSLXi0iAud3iJ3ZqWjY6EGg0DYypSWNVSh0wd72KMeuzt1IqBUM1NVoZYnofA+ObxFGNxkQHAo2mDszZrgtFJEdE9ojIeHO/xZQc+EZENorIJhG53TzsTszZ4iLSC7geeEwp5QZDMkQp9b6Z9j0zvUbTZOgmqUZTN/8G9AOuAaKBL0UkC0PyIwFIxVCP/RZYbh4zBGOWL0AKsEsZImK1kQOk+8Vzjaae6BaBRlM3N2AonLqUUieA/8W4cd8ArFJKuZVSxzEkLirpAhTUJ3MzQDhEJKyB/dZo6o0OBBpN3fha7KOuRUDKMLRgwNCquUZE6rrWAoHyH+GbRtMg6ECg0dRNFjDeXCikI8ZykNnAZxiL4lhEpBPGsoaVfAskAihj/YjtwDwvJcveIjLW3I4CCpRSFY11QhpNdXQg0GjqZi2GCuTXwCfAb8yuoDUYCp85wCsYq1sVmce8T9XAMBVjkaJ9IrIHY+2ESr39YcAm/56CRlM3Wn1Uo/mRiEioMlavisJoJQxRSh03NeS3mp99DRJX5vEuMFtdem1tjcZv6LeGNJofz0ZzAZEA4L/NlgJKqTIReQJjXdkffB1sLrjzng4CmqZGtwg0Go2mjaPHCDQajaaNowOBRqPRtHF0INBoNJo2jg4EGo1G08bRgUCj0WjaOP8PZvQ3RytuuocAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "### 最佳超参数\n",
    "print(\"Best score: %f using %s\" % (-grid.best_score_, grid.best_params_))\n",
    "\n",
    "# plot CV误差曲线\n",
    "rbf_test_means = -rbf_grid.cv_results_[ 'mean_test_score' ]\n",
    "rbf_train_means = -rbf_grid.cv_results_[ 'mean_train_score' ]\n",
    "\n",
    "n_Cs = len(Cs)\n",
    "number_gamms = len(gammas)\n",
    "\n",
    "\n",
    "rbf_test_scores =  np.array(rbf_test_means).reshape(n_Cs,number_gamms)\n",
    "rbf_train_scores = np.array(rbf_train_means).reshape(n_Cs,number_gamms)\n",
    "\n",
    "# plot results\n",
    "print(rbf_test_scores)\n",
    "print(rbf_train_scores)\n",
    "\n",
    "x_axis = np.log10(Cs)\n",
    "for i, value in enumerate(gammas):\n",
    "    plt.plot(x_axis, rbf_test_scores[:,i], label= 'test gamma:'+ str(gammas[i]))\n",
    "    plt.plot(x_axis, rbf_train_scores[:,i], label=  'train gamma:'+str(gammas[i]),linestyle='dashed')\n",
    "\n",
    "plt.legend()\n",
    "plt.xlabel( 'log(C)' )                                                                                                      \n",
    "plt.ylabel( 'Logloss' )\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "参数C：\n",
    "训练误差，随着C的增大，越来越小\n",
    "测试误差：先减小达到最佳模型对应的C=1时，C再增大，模型出现过拟合现象，测试误差反而增大\n",
    "\n",
    "参数gamma:\n",
    "当gamma设得太小时，会造成平滑效应太大，训练误差和测试集上的误差就比较大。\n",
    "当gamma设得合适时，则训练误差和测试误差都比较小  \n",
    "当gamma设的太大，训练误差达到最小，但不管C的取值是多少，测试误差都很高，发生过拟合"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
